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Dr. William E. Christopher 

Commissioner 

325 West Washington 

Indianapolis, IN 46204 

Dear Dr. Christopher: 

Phi Delta Kappa is pleased to forward this recommended set of evaluation criteria for vocational and tech- 
nical education in Indiana. Consistent with the commission's plan that calls for the evaluations to be out- 
comes oriented, these criteria are designed to be used in evaluating the effectiveness of Indiana's vocational 
and technical education delivery system. Phi Delta Kappa is pleased to have been involved in this effort, 
and we feel that this approach has national potential as a model process for evaluating these programs. 

First, we extend our appreciation to the State Advisory Committee. Invaluable advice was provided in 
the consultations about membership on the National Advisory Committee. Even though this group did 
not have the opportunity to review the recommended criteria in detail, the State Advisory Committee did 
have the oppormnity to discuss the criteria in an early stage of their development. As noted in the discus* 
sion on implementation in the report, the criteria would be field-tested and used on a demonstration basis 
before they are adopted and installed on a formal basis. The state group's level of involvement in this 
smdy may have been somewhat limited, but the recommended process assumes a heavy level of involve- 
ment at the program provider level as the criteria are being implemented. 

We also wish to express our appreciation for u»e cooperat-on and assistance provided by the staff of the 
commission and the three state agencies with responsibilities in this area. Staff members were accessible 
and willing to provide information and respond to inquiries as well as clarify expectations. Staff members 
were consulted at various points during the study, and, even though the final product may include criteria 
that were questioned by some staff rnemb^ rs, those comments were given consideration. Indiana's voca* 
tional and technical education delivery system is continuing to develop, and we are confident that staff 
members of the commission and the various state agencies are cognizant of the variety of interests and 
pressures that were considered during the course of this study. 

Special appreciation is accorded to the National Advisory Comminee; this broadly representative group 
brought a national perspective to the study* They provided assistance in finalizing the overall design of 
the study, but their greatest contribution was in developing the concept of effectiveness criteria as the 
cornerstone of this recommeiided evaluation system. In this manner, Indiana has the opportunity to pro- 
vide national leadership in the evaluation of vocational and technical education programs. 

This effort has been consistent with Phi Delta Kappa's goals of research, service, and leadership, and 
we hope that this report will be of value to the commission in its continuing efforts to improve vocational 
and technical education in Indiana. 
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FOREWORD 



This report's focus on the use of effectiveness criteria in evaluating vocational and technical educa- 
tion in Indiana is another step in Indiana's continuing efforts to be a national leader in vocational 
and technical education. The state has an historical commitment to developing and maintaining a 
responsive set of quality vocational and technical education programs to ensure the continued de- 
velopment of Indiana's diverse economy. Consistent with those historical goals, this recommended 
evaluation system has several distinctive features that are designed to reflect the values of the state 
and to ensure that continued attention will be given to development of die state's human, intellectual, 
and material resources. The forward-thinking di; tinctiveness of this report is evidenced in the con- 
tinuing emphasis that is given to the following considerations. 

Rather than adapting traditional evaluation models, this report's study design and the resulting 
evaluation system represent a departure from tradition. In this manner, they are consistent with 
two distinctive features of the Indiana setting: 

a. Indiana has been a leader in the development of a human resource system for planning and 
delivering vocational and technical education; the state's framework was in place l»efore the 
federal government took similar actions in reauthorizing the Perkins Vocational Education Act. 

b. Indiana has made the decision to use outcome or effectiveness criteria in evalrating 7ocatio:i- 
al and technical education. This performance-driven model has been adopted ev^n though 
historical evaluati(»n efforts have been input-oriented and fiscal allocations have been enio'Jment- 
driven. Under this system, programs are to be evaluated on the basis of ib^ employability 
and work-success of those who complete the progi-ams. The policy choice is irregular and 
may be incompatible with current management and funding practices, but it illustrates that 
productivity and employability are not necessarily enrollment-driven or academic goals. 

Periodic ch^ges have been, and may continue to be, made in Indiana's strucmral system for 
providing vocational and technical education; however, this report's evaluation system has been 
designed to focus on the efficacy of local vocational and technical education programs that likely 
will remain relatively stable. 

Stability may be sought, but it is not always desirable. To increase the likelihood that the evalua- 
tion criteria will remain relevant and current, this report's evaluation system assumes frequent 
review and revision of the process and the criteria as well as adaptation to different situations. 
This continuing renewal process offers its own style of stability and distinctiveness. 

This report's emphasis is consistent with recent professional literature. Much has been written 
about the criteria and the process of program evaluation, but little has been written about what 
should be done in consequence to the evaluation. This report stresses the importance of follow- 
up to the evaluation in the recommended implementation process. However, from a different 
perspective, the challenge for state-level leaders and program administrators may be to recon- 
cile the emphasis on effectiveness criteria for a performance-driven program in a vocational and 
t^hnical educational system whose strucmre and design assumes an enrolhnent-driven program. 

By focusing on program effectiveness, the report's intent is that attention will be given to pro- 
gram imputs when low program performance suggests the need for program improvements. In 
designing this evaluation system, a high value was placed on minimal reporting requirements 
and the capacity to adapt the evaluation system to different situations. 
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EXECUTIVE SUMMARY 



Interest in evaluating vocational and technical education has increased with changes in the federally 
legislated evaluation requirements and the national interest in economic development. Rather than 
suggesting that vocational and technical education focus more on acquisition of specific skills, recent 
reform recommendations have stressed mastery of basic academic skills and preparation for multiple 
jobs during the working years. 

Indiana's three-level system for delivering vocational and technical education is comprehensive 
and should facilitate program plaiming, delivery, and coordination. The coordinating body is the 
Indiana Conrnnission on Vocational and Technical Education* The state-level administering agencies 
are the Commission on Higher Education, Department of Employment and Training Services, and 
Department of Education. Programs are conducted by 302 school corporations, 49 area vocadonal 
districts, 7 public postsecondary institutions, independent colleges and universities, private proprietary 
schools, conmiunity-based organizations, and other agencies. 

Within this context, the Indiana Commission on Vocational and Technical Education awarded Phi 
Delta Kappa International (PDK) a contract to develop evaluation criteria for vocational and techni- 
cal education. PDK then contracted for a description of the current Indiana evaluation system, a review 
of the research on evaluation of vocational and technical education, and a study of practices used 
to evaluate vocational and technical education in other states. National and state advisory commit- 
tees were appointed to assist in the project. Consistent with the current delivery system, criteria have 
been developed for the commission, state agencies, and local program providers. 

To maximize local flexibility in conducting and evaluating programs and to recognize that differ- 
ent approaches can be used to achieve similar goals, the central focus of these recommended criteria 
for evaluating vocational and technical education in Indiana is on program effectiveness. The histori- 
cal contention has been that program outcomes cannot be measured effectively because of a lack 
of agreement on goals and variations in inputs and practices. However, current evaluation practices 
assume that program inputs and processes can vary but that standard criteria can be used to measure 
program effectiveness and outcomes. 

Foi* each effectiveness criterion, process criteria have been identified that can then be used by agencies 
in their internal assessment of the extent to which effectiveness criteria have been attained. Additions 
or deletions in the process criteria may be necessary to ensure their applicability and relevance. In 
designing this evaluation process, the goal has been to find the balance between minimal reporting 
for the agency or program provider and the information required to evaluate the programs. 
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PARTI 
Introduction 

Vocational and technical education in America's schools traces its history to 1917 with the p»;^sage 
of the federal legislation under the Smith-Hughes Act. As the nation entered a period of industrial 
expansion, emphasis was placed on the need for the worker to develop specific occupational skills. 
From this beginning, vocational and technical education has expanded into a multi-agency delivery 
system involving secondar>' schools, public and private postsecondary schools, job specific short- 
term training programs, and collegiate institutions. The current interest in the relationship between 
vocational and technical education and economic development and international competitiveness is 
a continuation of the historical rationale for these programs. 

Ne^ Directions in Vocational and Technical Education 

As with other aspects of education, social and economic changes and the forces of education reform 
are impacting on vocational and technical education. The dramatic changes in the workplace and 
the possibilities of multiple jobs in a person's lifetime have resulted in an acceptance of the premise 
that vocational and technical education should emphasize student attainment of broad transferable 
skills and the importance of relating vocationa; and general education (Kadamus, 1986). The assumption 
is that a common set of basic skills and work habits is needed for a person to function in today's 
and tomorrow's society (Lindcman, 1984; Investing in Our Children, 1985). 

The need for vocational and technical education to be responsive to changing conditions has been 
noted from a variety of sources. In a summary of a series of conferences on trends and implications 
for educational leaders, vocational and technical education was admonished to project needs beyond 
additional funds (Clodi and Jacobson, 1989). Areas of concern included: 

• development of alliances 

• exploitation of successes 

• greater use of technology in instruction 

• reform of the teaching/learning process 

• responsiveness to constituents 

• adoption of strategic planning 

Observers contend that vocational and technical education is facing tt)e choice of adapting to changing 
economic conditions and societal expectations or entering into a period of decline. Kadamus and 
Daggett (1986) have identified six major changes that should be considered in planning for vocation- 
al and technical education programs: 

• De-emphasis ot specialized training and an increased emphasis on broad transferable job skills 
as well as job-specific skills. 

• Shift from large expenditures on specialized equipment to funds for staff development and 
equipment that help students develop an understanding of concepts. 

• Change in program emphasis from trades and industry to information and service-related oc- 
cupations. 
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• New linkages with business and industry. 

• Development of programs to raise the pi^lic*s awareness of vocational opportunities. 

• Goaer relationships with general (academic) education to increase cuiricular and program coor- 
dination. 

Different types of vocational and technical education programs are emerging in response to this 
desire to broaden the context of vocational and technical education. One example of the merger of 
general and vocational and technical education into an integrated program is illustrated by the 2<f 2 
or 2 2 4' 2 tech-prep programs found in California, Indiana, Michigan, New York, North Carolina, 
Ohio, and Virginia. In these programs, student outcomes are built around a cooperatively selected 
core of ccnnpetencies specific to a selected occupational area. The 2 2 refers to a student being 
in a vocational and technical education program that begins during the last two years of secondary 
school and continues through two years in a postsecondary institution such as a vocational school 
or community college. Under the 2 2 2 plan, the program is extended for an additional two years 
into a baccalaureate institution and concludes with a bachelor's degree. Exit requirements for these 
programs include a mastery of job and academic employ ability skills (Walker, 1989). 

Evaluation of Vocational and Technical Education 

In the past two decades, the interest in evaluation of vocational and technical education programs 
has increased. With the enactment of the Education Amendments of 1976 by Congress, a vocational 
education data and reporting system was established. The justification was that the lack of compara- 
ble data across states made federal policy decision making an almost impossible task. However, the 
data reporting system did not become fully (^rational because the system failed to provide the desired 
quality or quantity of data. This earlier emphasis on comparable data was followed by an emphasis 
on educational outcome data when the legislation was reauthorized in 1984 as the Carl D. Perkins 
Vocational Education Act. Emphasis was placed on both occupational skills and academic founda- 
tions. A secondary reason for the interest in evaluation at the program provider level was that stu- 
dents seeking federal postsecondary student assistance had to be in a program accredited by either 
a voluntary accrediting agency or a state agency. 

Evaluation also has been emphasized in job training programs fiinded through die U.S. Depart- 
ment of Labor. Emphasis has been placed on performance standards and program exit knowledge 
and skills. As with vocational and technical education programs, job placement data also have been 
an integral part of the information base for evaluation. 

With the emphasis on outcomes, products, and effectiveness, the temptation has been to place in- 
creased emphasis on standardized test results for the measurement of student progress. Rather than 
relying on standardized tests that may not be relevam to the particular situation, a consortium of 
state educational agencies is developing task lists specific to occupations. These items then will be 
subjected to a task analysis and subsequently used as the basis for development of a computerized 
catalog of test items to be shared across states and local school dUtricts (Gourley, 1989). 

Even though the educational outcome focus has been in effect for almost five years, only a few 
states have developed evaluation systems for meeting the federal requirement. The principal issues 
are related to what is to be evaluated, what evaluation system or model should be used, and which 
level - state or local - should conduct the evaluation. State and local agencies also have been slow 
to respond because changing federal priorities have contributed to a variety of definitions, changes 
in program focus, and additional target populations. 

The shift from traditional input/process evaluation to outcome evaluation has forced state-level 
decision makers to address the issues of the defmition aua ourpose of vocational and technical educa- 
tion. Both state and local agencies are seeking ways to measure and report student progress and link- 
ages between programs and on-the-job tasks. The principal trends appear to focus on transferable 
and specific occupational skills, reliance on individualized programs, evaluation as a local function. 
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develo(MDent of state technical assistance programs, and an expanded base for types of programs 
and program participants. 

The challenges in designing an evaluation system are to secure sufficient information to meet the 
demands for accountability from governmental agencies, employers, and the general public, but to 
refrain from gathering unnecessary information. Stated differently, in making decisions, '"need to 
know" must be contrasted with "nice to know." However, evaluation has a function beyond provid- 
ing state and federal agencies with nmnitoring data. The maximum contribution at the pr^ 
level is attained when the program evaluation process provides the information base for decisions 
about program focus, program improvement, and program changes. In this context, program evalu- 
ation information is the primary source for program policy decisicms. 



Development of Evaluation Criteria 

Widiin the context of these national conditions, the Indiana Commission for Vocational and Techni- 
cal Education (CVTE) is developing an evaluation system for vocational and technical education in 
Indiana. The commission has contracted with Phi Delta Kappa International (PDK) '*to identify and 
validate evaluation criteria to be used to assess the overall effectiveness of vocational and technical 
education, including training and employment." Recognized national experts on vocational and tech- 
nical education and the process of evaluation have been involved in the development of the evalua- 
tion criteria for Indiana. Sources have included the body of literature and research on evaluation 
as well as the practices in selected states. 

The following secti(Mi contains a set of premises and assumptions about the purposes and evalua- 
tion of vocational and technical education in Indiana; this nuiterial provided the conceptual frame- 
work for much of this report. Following the premises, a brief description is presented of the existing 
administrative structure for vocational and technical education in Indiana. 

In response to the project contract, individual sets of effectiveness criteria are presented in pan 
n for the program delivery sites, state administering agencies, and CVTE. (Process criteria and ef- 
fectiveness criteria are listed in the appendix.) The process used for validating die criteria is dis- 
cussed in part ID. Recommendations for implonentadon may be found in pan IV. The project agreement 
stipulated that PDK was to review and summarize current evaluation systems being used in Indiana; 
a summary of this repon is presented in part V. (See accompanying research report no. 1 for the 
complete report.) The response to the requirement for a review of the research literature and related 
documents on evaluation of vocational and technical education is summarized in part VI; the com- 
plete report is in accompanying research reports nos. 2 and 2a. As stipulated by the project contract, 
a summary of the evaluation practices used in selected states is reported in part VII. (See accom- 
panying research reports nos. 2 and 2a.) 



Premises and Assumptions 

CVits long-range plan and comments of the National Advisory Committee were the principal sources 
for the following premises and assumptions about vocational and technical education in Indiana. These 
statements guided the development of the criteria for evaluating the vocational and technical educa- 
tion delivery system. 

Goals of Vocational and Technical Education. 

The goals of vocati(mal and technical education in Indiana are to 

• provide a quality w(vkforce by preparing citizens for both enqjioyment and further education; and 

• provide citizens with employability skills and retraining capacities. 
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Functions of the Commission, 
The fiinctioas of the commissioD are to 

• promoce the development of vocational and technical education delivery systems in Indiana 
that can cope with diversity and respond to economic and demographic changes; 

• establish system and program evaluation as a process of continuous self-improvement that 
results in the enhuioement of programs and servia'^ .o respond to employment and training 
needs and stimulate economic development; and 

• distinguish die job preparation mission of vocational and technical education from the gen- 
era! education mission of regular education and translate these differences into an action 
plan. 

Partnershyfs. 

The development of a comprehensive system of vocational and technical education for Indiana in- 
volves the creation and maintenance of partnership programs in which current and potential employ- 
ers and traditional vocational and technical education delivery agencies work together with public 
and private agencies in joint utilization of resources to provide programs in both traditional and non- 
traditional settings. 

Responsive Detivery System. 

The maintenance of a responsive vocational and technical education delivery system in Indiana re- 
quires the periodic review of delivery agencies and adaptations as well as modifications in response 
to changing needs. The system should promote change, stimulate local comjnitment, improve pro- 
gram quality, and encourage the development and attainment of performance standards. 

Evaluation System. 

The recommended evaluation system for vocational and technical education in Indiana assumes a 
three-tier organizational structure and evaluation process. Effectiveness criteria and related process 
criteria wiU be used to evaluate the effectiveness of: (1) the services and programs being furnished 
by local program providers; (2) the operations of state agencies; and (3) the operations of the com- 
mission and the overall vocational and technical education delivery system in the state. Evaluation 
results from each level then can be reviewed to identify changes that might be made to improve pro- 
grams and services at the program provider level and to identify implications for short and longterm 
state and local public-policy decisions about vocational and technical education. 

Evaiuation Criteria. 

The primary focus of the effectiveness criteria is on the extent to which the program has been effec- 
tive in meeting the state and local objectives for vocational and technical education. Related process 
criteria will provide information about the extent to which the goals or targets of tte effectiveness 
criteria have been attained and serve as indicators of stq>s that might be taken to improve programs. 
These process criteria are viewed as illustrative, and additions or deletions to the process criteria 
may be necessary to ensure applicability and relevance. 

Staff from the commission and state agencies should work with program providers to review and 
modify the process criteria and identify iq>propriate indicators and performance standards for each 
effectiveness criterion. This cooperative process for identifying and developing indicators and per- 
formance standards should improve the base of support for the evaiuation process. The goal should 
be to adapt the evaluation process to individual sit>iations. For e:<ampb, an indicator might be ap- 
plicable to the entire state, but the related relevant data and performance standards might vary by 
program, geographical area, or characteristics of the program participants. 
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Phased ImpUmenUHion. 

The evaluation program should be phased rather than immediately implemented. The connmission 
and the state agencies should work with program providers to identify the indicators and perfomn- 
ance standards for each criterion. Each agency should have the responsibility to develop an implemen- 
tation plan for its evaluation program. 

Vocational and Technical Education in Indiana 

Under Indiana's administrative structure for vocational and technical education, CVTE functions as 
an oversight and leadership agency for vocational and technical education in Indiana. Contacts with 
those local school districts and other agencies and institutions that conduct programs are through 
three state-level agencies: 

Indiana Commission for Higher Education (CHE) 
Indiana Department of Education (DOE) 

Indiana E>epartment of Employment and Training Services (DETS) 

CHE is the state-level agency for vocational and technical education programs conducted by the 
following service provider sites: Indiana Conunission on Vocational and Technical Education. Vin 
cennes University, Ball State University, Indiana State University, Indiana University, Purdue Univer- 
sity, and the Southern Indiana Universities. DOE is the state-level agency for vocational and technical 
education programs conducted by the 302 local school corporations and 49 area vocational districts 
in Indiana. DETS is the state-level agency for specific job training programs; the actual programs 
are conducted by conmiunity-based organizations and other local entities, including some of the in- 
stitutions under the jurisdiction of CHE and DOE, 

In the following section, separate sets of effectiveness criteria have been developed for the pro- 
gram providers, state agencies, and commission. The developmental process has assumed that each 
set of criteria would include the stamtory and programmatic responsibilities applicable to that insti- 
tution or agency and that each set of criteria could be used independently of the other two. This 
has resulted in some criteria in one set being somewhat similar to those in another set, but an effort 
has been made to state each criterion in a manner that makes it specifically applicable to its level. 
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PART II 
Criteria for Evaluating 
Indiana Vocational and Technical Education 



The following effectiveness criteria for evaluating vocational and technical education in Indiana are 
presented in three groups - local program provider, state agency, and commission. One or more 
process criteria arc presented for each effectiveness criterion in the appendix. Process criteria should 
be viewed as illustrative; the assumption is diat deletions, additions, and adaptations will be made 
when appropriate. Program administrators would select process criteria that would be used to indi- 
cate the extent to which each of the effectiveness criteria had been attained. 

Some effectiveness criteria appear in all three sets, but the emphasis varies depending on the or- 
ganizational levers specific responsibilities. The commission has greater leadership and planning respon- 
sibilities. The state agencies have greater monitoring and technical assistance responsibilities, and 
program providers have greater delivery and evaluation responsibilities. 

In this report, the term "commission's long-range plan" refers to Education and Jobs for Indiana's 
Future, the long-range plan for a comprehensive vocational program in Indiana, as approved by the 
commission on July 14, 1988. This plan, agency plans, and local plans are different from those re- 
quired by the U.S. Department of Education to participate in federal programs. 



Effectiveness Evaluation Criteria for Program Providers 



The following list of effectiveness criteria for program providers was developed as a response to 
task 4 in the project contract. Program providers include the state's 302 school corporations, 49 area 
vocational districts, 7 public postseccHidary institutions, independent colleges and universities, pri- 
vate proprietary schools, community-based organizations, and other agencies. Criteria for evalua- 
tion at this level have been developed from the material submitted by the various contractors. In 
this document, the criteria have been divided into the following areas: 



These evaluation criteria are designed to facilitate tracking the progress of program delivery sites 
in implementing those strategies outlincl in d>e state and local long-range plans and to permit refme- 
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ment of q)ecific evaluation criteria as strategies are implemented. Hrocess criteria for each effective- 
ness criterion are listed in the appendix of this report. They may be used by each program delivery 
site in flff«ff«if'*!g the extent to which an effectiveness criterion has been attained and in determining 
responses to identified problems. 

1.0 Structure 

EJSpectiveness Criterion 

1.1 Philosophy and Organization - Program provider's philosophy statement and adminis- 
trative organization are relevant to the immediate service area, reflect current conditions, 
and are consistent with and supportive of current programs and documents from the com- 
mission and the relevant state agency. 

2.0 Planning 

Effectiveness Criterion 

2.1 Local Comprehensive Plan — Program provider's c*.trrent and propo'^'^ programs are 
consistent with and supportive of the program provider's comprehensive long-range plan 
and long-range plans of the conmiission and the relevant state agency. 

3.0 Diagnostic Assessnsent 

Effectiveness Criterion 

y.l Diagnostic Assessment - Program participants and staff use diagnostic assessment in- 
formation in making decisions about program offerings and individual program selections. 

4.0 Program Design and Delivery 
Effectiveness Criteria 

4.1 Program Design Decisions - Delivery systems accommodate the special needs of pro- 
gram participants and provide for exposure to and participation in actual work settings. 

4.2 Curriculum Decisions - Program provider administrators use a systematic and compre- 
hensive approach in making curriculum decisions and use data related to the inunediate 
service area, program participants, and the goals and functions of related agencies and 
institutions. 

4.3 Instructional Decisions - Program components are designed to make participants aware 
of career opportunities, reinforce positive work attitudes, attain basic skills, de*elop em- 
ployability skills, and learn specific occupational skills and good safety practices. 

4.4 Staff Development — Program provider takes steps to ensure that the staff has and will 
continue to develop the qualifications and skills to provide the selected programs. 

4.5 Funds — Required human and material resources are appropriately allocated within and 
among programs. 

5.0 Participant Services 

Effectiveness Criteria 

5.1 Program Access - Broad access is provided to the handicapped, disadvantaged, and 
minorities. 

5.2 Program Services - Program provider selects and provides services to address the needs 
of current and projected program participants. 

5 J Program Breadth - Program participants have access to different choices within the pro- 
gram provider's cycle of offerings. 
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6.0 Partnerships 

Effectiveness Criteria 

6.1 Partnership Programs — Program provider develops and maintains partnership pro- 
grams between and among business, industry, labor, government, and education. 

6.2 Technology Education Curriculum — Program provider involves business, imiustry, 
labor, government, and education in the design, development, and implementation of 
the technology education curriculum. 

7.0 Facilities and Equipment 

Effectiveness Criterion 

7.1 Facilities and Equipment - The facilities and equipn)ent meet acceptable standards 
of safety, adequacy, adaptability, accessibility, and public healui. 

8.0 Participant Outcomes 

Effectiveness Criteria 

8.1 Demonstrated Competency - Program completers can demonstrate the competencies 
that are the i^oals of the program and the expectations of employers. 

8.2 Job-Related Tasks - Program completers apply their knowledge and skills in securing 
and keeping a job. 

9.0 Program Accountability 

Effectiveness Criteria 

9.1 Enrollment/Completion Targets — Program participant enrollment levels and comple- 
tion rates are at, or exceed, the established standard. 

9.2 Related Employment - Program completers are employed in the jobs for which they 
were trained. 

9.3 Gainful Employment - Program completers are employed in gainful occupations. 

9.4 Salaries - Salaries paid to program completers working in a training-related job ex- 
ceed those for persons who did not complete the training. 

9.5 Job Stability - Program completers exhibit a specified level of employment stability. 

9.6 Employee Satisfaction — Program completers indicate that they are satisfied with the 
content, experiences, and outcomes of the training program, and have job satisfaction. 

9.7 Employer Satisfaction - Employers are satisfied with the on-the-job performance of 
the persons employed from the program. 

9.8 Postprogram Education - Program provider has foUow-up information to determine 
the number of program completers who complete high school and/or pursue additl^'^al 
formal education. 

10.0 Fiscal AccountabiUty 

Effectiveness Criterion 

10.1 Fiscal \udits - Financial audit reports indicate that fiscal procedures are in compli- 
ance with applicable local, state, and federal rules. 
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11.0 Cost-Effectiyeiiess 

Effectiveness Criterion 

11.1 Cost-Effectiveness - Given the program objectives and compared to the cost of simi- 
lar programs, cost data per program enroUee, completer, and placement irdicate that 
the program is cost-effective. 

Effectiveness Evaluation Criteria for State Agencies 

The following list of effectiveness evaluation criteria for the state agencies lias been developed as 
a response to task 4 in the project contract. The state-level administering agencies for vocational 
and technical education are the Commission on Higher Education, Department of Employment and 
Training Services, and Department of Education. The principal sources for these criteria were the 
conmiission's long-range plan, existing evaluation < riteria, rrlated statutes, and commissioned studies. 
The criteria have been divided into nine broad areas: 

1.0 Structure 
2.0 Long-Range Plan 
3.0 Agency Leadership 
4.0 Fiscal Accountability 
5.0 Technical Assistance 
6.0 Program Evaluation 
7.0 Articulation 

8.0 Professional Personnel Development 
9.0 Partnerships 

These evaluation criteria are designed to facilitate tracking the progress of state agencies in im- 
plementing those strategies outlined in the commission's long-range plan and to permit refmement 
of specific evaluation criteria as strategies are implemented. Process criteria for each effectiveness 
criterion are listed in the appendix of this report. They may be used by the agency in assessing the 
extent to which an effectiveness criterion has been attained and in determining responses to identi- 
fied problems. 

1.0 Structure 

Effectiveness Criterion 

1.1 Statement of Purpose and Administrative Organization — Agency's statement of purpose 
and administrative organization are relevant to the programs, reflect current conditions, 
and support local programs. 

2.0 Long-Range Plan 

Effectiveness Criteria 

2.1 Consistency with Commission's Long-Range Plan - Agency decisions on delivp.y sys- 
tems and programs for vocational and technical education are consistent with the detailed 
provisions of the conunission's long-range plan. 

2.2 Goal Relevancy — The goals in the agency's long-range plan for vocational and technical 
education are being used to guide policy decisions at the agency and local levels. 

3.0 Agency Leadership 

Effectiveness Criteria 

3.1 Coordination - Agency works effectively with other state agencies, conunissions, and 
organizations that supervise or administer vocational and technical educatior4 programs 
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in the development of plans or programs to improve the coordination of vocational and 
technical education in Indiana. 

n«,^ in ndi« stale aaendesvothiBSponsihihticsr 

3 J Pnrfta.ion.IRel«io«ship.- Agency perso™i«orkwid,progran.provito 

U) improve the wtaiinistration deUvery of programs. 
3 4 Program Guidelines - Agency prnvides progr^n providers wi* bro«l guidelines, in- 

Son. and resources for curriculum development «k1 updating. 
3.5 I^ogram Provider Long.R».ge PI». Review - Agency provides program providers -vim 

feedback on their local long-range plans. 

4.0 Fiscal Accountability 

Effectiv«uss (^""'^ ^5 ^ J that encourages programmat- 

ment and attainment of performa^icc standards. 

4.2 Sufficient Fu«is - The level of funding ^<^^^:^ :!Z^;'Z'^Z 
is sufficient to provide adequate financial support for the necessary numocr anu 

ance of programs to meet state goals. 

4.3 Funds AUocation - Agency dispense, funds for ^"^'^'^ 

««m in a mam«r consistent with the statutory requirements of the Indiana General As 
sembly, federal regulations, and the commission's recommendanons. 

5.0 Teclinical Assistance 

f^^^Uvd T^nical Xssistance - Agency provides technical assistance on p^gram 
pS Ls^on, aiministration, and evaluation to local pr,gram providers. 

6.0 Program Evaluation 

fTt^l^vZion - Agency uses a systematic and comptchensive approach in evalu- 
ating vocational and technical education programs. 

^ -» iT» r,«tJnn nf Data - Aecncv's data base is used in making both program and public- 
"d^c" l^lut vSal and technical education at the agency and local pro- 

gram provider levels. 
7.0 Articulatioii 

fftS.^ WUon - Prog,™. p«.icip»«s have only minimal «dunda«,. l^Uc^ 
Sr^Sp\^P«*l«^™«»g the transitions^ 

^SJy «Kad(»al and technical education programs and college degree programs. 
8.0 Professional Personnel Development 

SXuion - Tl- agetKy guideUnes for voc..^ 
■ tification are consistent with the commission's recommendations. 
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8.2 Sufficient Staff - The number of qualified vocational and technical education teachers 
is sufficient to staff the programs in the state. 

8.3 Teacher Skill Improvement - Agency works with piogram providers in implementing 
ttie commission's long-range |^ for inootptxating updated tccluiological content and teach- 
ing skills into vocational and technical education programs. 

8.4 Tax Incentives for Teachers - Where applicable, the agency makes legislative recom- 
mendations to the commission for a professional development credit system and tax in- 
centives for teachers. 

9.0 Partnerships 

Effectiveness Criteria 

9.1 Partnership Programs - Agency develops and maintains partnership programs between 
and among business, industry, labor, government, and education. 

9.2 Technology Education Curriculum - Representation from business, industry, labor, 
govermnent, and education assists in the design, development, and implementation of 
the technology education curriculum. 

System Effectiveness Evaluation Criteria for 

Indiana CommissiOii on Vocational and Technical Education 

The following list of effectiveness evaluation criteria for the overall systems evaluation of the Indi- 
ana Commission on Vocational and Technical Education has been developed as a response to task 
5 in the project contract. The Indiana Commission on Vocational and Technical Education is the 
state-level coordinating body for vocational and technical education. The principal sources for these 
criteria were the commission's long-range plan, related statutes, and commissioned studies. The criteria 
have been divided into nine broad areas: 

1.0 Long-Range Plan 

2.0 Duties of the Commission 

3.0 Fiscal Resources 

4.0 Program Delivery 

5.0 Articulation 

6.0 Professional Personnel Development 
7.0 Partnerships 
8.0 Technical Assistance 
9.0 Vocational and Technical Education 
Evaluation Criteria 

These evaluation criteria are designed to facilitate tracking the progress of the conunission in im- 
plementing those strategies ouUined in the conmiission's long-range plan and to permit refinement 
of specific evaluation criteria as strategies are implemented. Process criteria for each effectiveness 
criterion are listed in the appendix of this report. They may be used by the commission in assessing 
the extent to which an effectiveness criterion hits been attained and in determining responses to iden- 
tified problems. 

1.0 Long-Range Flan 

Effectiveness Criterion 

1.1 Commission's Long-Range Plan - Commission adopts and updates a long-range plan for 
the provision and continuous in^KOvemem of vocational and technical education in Indiana. 
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2.0 Duties of the CommisskMi 

Effectiveness Criteria 

2.1 Eoactnient of Recommendations - The Indiana General Assembly enacts the conunis- 
sion's recommendations concerning the development, dq>Ucation, and accessibility to em- 
ployment training and vocational and technical education on a regional and statewide basis. 

2.2 Coordination - Commission works effectively with state agencies, state commissions, 
and community-based organizations that supervise or administer vocational and technical 
education programs in the development of plans or programs to improve the coordination 
of vocational and technical education in Indiana. 

2.3 Federal Funds - Commission receives and allocates the federal funds available for voca- 
tional and technical education 

2.4 Communication - Conunission has systematic and purposeful conmiunication channels 
with business, industry, and labor organizations that operate vocational and technical edu- 
cation programs. 

3.0 Financial Resources/Responsibilities 

Effectiveness Criteria 

3.1 Priorities and Goals - Commission allocates funds in a manner that encourages program- 
matic change stimulates local investment, improves program quality, and fosters the de- 
velopment and attainment of performance standards. 

3.2 Funds Allocation — The Indiana General Assembly enacts the commission's recommen- 
dations for allocating funds for vocational and technical education programs. 

3.3 Private-Sector Tax Incentives - The Indiana General Assembly enacts legislation to provide 
tax incentives and tax credits to encourage involvement of private-sector firms in voca- 
tional and technical education programs. 

3.4 Foundation - Commission has established and maintains a nonprofit foundation for the 
improvement of vocational and technical education in Indiana. 

3.5 Fiscal Audits - Financial audit reports indicate that fiscal procedures are in compliance 
with applicable local, state, and federal rules. 

4.0 Prognun Delivery 

Effectiveness Criteria 

4.1 Program Provider Evaluation - Commission's evaluation and oata system provides and 
uses information to improve the effectiveness of individual program providers. 

4.2 Statewide Program Evaluation — Commission's evaluation and data system provides and 
utilizes statewide information on specific occupational training programs to improve the 
statewide effectiveness of these progranL«. 

4.3 System Program Effectiveness — Commission's evaluation and data system provides and 
utilizes information to improve the effectiveness of the state's comprehensive vocational 
and technical education delivery system. 

5.0 Articiilation 

Effectiveness Criterion 

5.1 Institutional Transition — Commission has taken the necessary steps to ensure that pro- 
gram participants have only minimal redundancy, logistical, or program-planning prob- 
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lems making the transition among and between secondai> and postsecondary vocational 
and technical education programs and college degree programs. 

6.0 Plrofesskmal Persmmel Development 

Effectiveness Criteria 

6.1 Teacher Certification ~ State system for certifying or recertifying vocational teachers 
is consistent with the commission's recommendations. 

6.2 Sufficient Staff - The number of qualified vocational and technical education teachers 
is sufficient to staff the programs in the state. 

6.3 Teacher Skill Improvement - Program providers implement the commission's long-range 
plan for incorporating updated technological content and teaching skills into teaching 
programs. 

6.4 Tax Incentives for Teachers - The Indiana General Assembly accepts the commission's 
recommendations for a professio:ial development credit system and tax incentives for 
teachers. 

7.0 Partnerships 

Effectiveness Criteria 

7.1 Partnership Programs - Commission develops and maintains partnership programs be- 
tween and among business, industry, labor, government, and education agencies. 

7.2 Technology Education Curriculum - Representation from business, industry, labor, 
government, and education provides assistance in the design, development, and implemen- 
tation of the technology education curriculum. 

8.0 Technical Assistance 

Effectiveness Criterion 

8.1 State-Level Technical Assistance - Local program providers receive technical assistance 
on program plaiming, instruction, administration, and evaluation. 

9.0 Vocational and Technical Education Evaluation Criteria 

Effectiveness Criteria 

9.1 Program Evaluation - Commission uses a systematic and comprehensive approach in 
evrjuating vocational and technical education programs. 

9.2 Utilization of Data - Conmiission's comprehensive data base is used to make both pro- 
gram and public-policy decisions about vocational and technical education at the state 
and local levels. 
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PART III 
Validation Process 

The study design provided for the use of a jury to validate the criteria that are the products of this 
study. The criteria were not field-tested because of tinie and fiscal constraints. As noted in the dis- 
cussion on implementation, t^ie assumption is that field-testing will take place during implementa- 
tion. A National Evaluation Pi-oject Advisory Committee served as the jury to validate these criteria. 
This seven^member conunitta; consisted of three nationally recognized researchers in vocational and 
technical education, a state higher education administrator with responsibilities related to these areas, 
a director of vocational and technical education from an urban local school district, and two voca- 
tional school administrators. 

Following the conunission's acceptance of PDK's proposal, the National Advisor' Committee met 
on four occasions. The first session was for orientation purposes. In the second meeting, the struc- 
nare of vocational and technical education in Indiana was reviewed, a preliminary set of evaluation 
criteria was examined, and research project progress reports were reviewed. 

The focus of the third meeting was on discussion of the summary of the research reports on evalu- 
ation of vocational and technical education and a brief report on current evaluation practices in Indi- 
ana and selected other states, format and type of evaluation criteria to be included in the reporr, and 
discussion of the content and outline of the final report. After considerable discussion at this meet* 
ing, the Nat^nal Advisory Committee determined that the focus of the criteria should be on the ef- 
fectiveness of the system for delivering vocational and technical education in Indiana. In the draft 
report, the focus was on effectiveness criteria* This approach assumes that the process criteria then 
would be used to provide evidence of attainment of a given effectiveness criterion or to identify the 
steps that might be taken to achieve that criterion. 

Prior to the fourth meeting, the National Advisory Committee received a draft of the final report 
including effectiveness and process criteria for three levels - program providers, state agencies, 
and commission. The purpose of the fourth meeting of the National Advisory Committ(.e was to 
serve as the jury to validate the criteria. The draft report included 23 effectiveness criteria for the 
program providers, 21 for the state agencies, and 21 for the commission, making a total of 65 criter- 
ia* The National Advisory Committee devoted most of its attention to a review and critique of the 
effectiveness criteria. 

After the critique by the National Advisory Committee, the report included 26 effectiveness criter- 
ia for the program providers, 21 for the state agencies, and 23 for the conmiission, making a total 
of 70 criteria. The principal changes made by the National Advisory Committee were to 

• include a separate section on partnerships in the set of criteria for program providers; 

• add a new criterion for state agencies and the commission concerning the use of funds to suo- 
port priorities and goals related to program innovation, cost-sharing, and program perform- 
ance and improvement; 

• add a new criterion for program providers related to breadth of offerings and student access 
to offerings; 

• add a new criterion for the conmiission on the availability of statewide cumulative data for 
specific training programs; 
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• delete a narrow criterion and broaden the context of remaining criteria on fiscal accountabili- 
ty for the state agencies; 

• strengthen the wording of various criteria so that they are more action oriented to ensure that 
the criteria have relevance to program improvement; and 

• make the effectiveness criteria more specific. 

Minor modifications were made in the process criteria to reduce redundancies among sets of criteria 
and to ensure that the effectiveness criteria and process criteria were appropriately related. How- 
ever, the process criteria were not reviewed in detail because the evaluation format assumes that 
the; are illustrative and will be deleted, supplemented, or modified to respond to differences in mis- 
sions, programs, and conditions. 

Members of the National Advisory Conunittee indicated acceptance of the effectiveness criteria 
as nKxlified in this report. The group also strongly supported the format that used effectiveness criteria 
with supporting optional process criteria. This format was viewed as progress toward attaining the 
optimal balance between providing the information required to evaluate the program and minimizing 
reporting requirements. 
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PART IV 
Implementation Process 



This discussion of the implementaticn process has been adapted from ""Recommended Steps for the 
Utilization of Criteria'* prepared by the Evaluation Center at Western Michigan University. (See ac- 
companying research reports nos. 2 and 2a.) In these steps, the assumption is that implementation 
of an evaluation program of this magnitude should be phased rather than immediate* For that reason, 
this proposed implementation process is presented for consideration by CVTE as it develops plans 
for implementing the evaluation process. Following the discussion of the implementation process, 
a list of data items that might be included in a program census is presented. In the last section, a 
summary of the implementation plan is provide to guide the commission in operationalizing the 
overall evaluation design. 

These recommendations were developed from a review of the literature on change processes, product 
utilization, general evaluation frameworks, and specific ev Juation processes that relate to vocation- 
al and technical education. The consensus appears to be that the first step in the evaluation process 
should be the development of a long-range plan. CVTE has developed a long-range plan for com- 
mission activities, and each state agency and program provider should develop a long-range plan 
related to its responsibilities and activities. These plans can then be used as guides in the develop- 
ment of respective implementation plans for the evaluation. 

Implementation Plan 

The focus of the following implementation plan is on activities at the program provider level. This 
choice was made because these delivery sites are the actual providers of vocational and technical 
education programs and services. CVTE and the state agencies have a supportive function; conse- 
quenUy, the steps in the plan are more related to program provider activities. 

This section recommends six steps for integrating a new set of evaluation criteria into an existing 
evaluation plan: 

1 . Create an awareness of the proposed criteria 

2. Demonstrate the application of the criteria 

3. Test the criteria in program provider settings 

4. Install the criteria 

5. Institutionalize the criteria 

6. Develop an evaluation follow-up system 

To clarify the reconamendation, a sample: description is provided for each step listed below. 
L Create an awareness of the proposed criteria. 

Members of evaluation conmiittees, teachers, administrators, governing board members, and other 
appropriate pec^le need to learn about the potential use and implications of the criteria. They need 
a clear understanding of the proposed criteria and the role of the criteria in the evaluation process. 

To acc(»iplish this step, state-level agency officials should develop appropriate written materials 
and orientation and training activities to create widespread awareness of the criteria and the existing 
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evaluation plan among the various constituents. Using these materials and activities the state-level 
agency should 

a. identify the various audiences and determine the level of their involvement in the criteria 
adoption process; 

b. determine the level of awareness necessary to insure effective involvement; 

c. conduct and evaluate the orientation and training activities; 

d. revise the criteria and/or the evaluation plan based on the results of the orientation and train- 
ing activities; and 

e. inforai the audiences about the revisions ard the reasons for these revisions. 
Example: 

The state-level agency identifies key legislators, the state school board association, the state asso- 
ciation of vocational educators, the state association of school administrators, and the state associ- 
ation of chambers of commerce as some of the awareness audiences. The agency informs these 
audiences about the proposed criteria and the existing evaluation plan. A 30-minute videotape 
presentation is developed, field-tested, revised, and presented to these audiences. Based on written 
evaluations of the video presentation, some criteria are revised to clarify their meaning. Copies of 
the revised video, with the reasons for these revisions, are mailed to each of the awareness par- 
ticipant agencies. 

2, Demonstrate the application of the criteria. 

Prior to field tesiing, a program is initiated to make constituents aware of how the criteria and the 
evaluation plan might work in actual practice. To take this step, the state-level agency should de- 
velop appropriate materials and demonstration activities designed to help the various constituents 
understand the criteria and the way that the evaluation plan might work in actua] practice. Using 
these materials and activities, the state-level agency should 

a. identify the audiences that should participate in demonstration activities; 

b. determine the extent of demonstration required to insure effective involvement; 

c. conduct and evaluate the demonstration activities; 

d. revise the criteria and/or the evaluation plan based on the results of these demonstration ac- 
tivities; and 

e. inform the awareness and demonstration audiences about the revisions and the reasons for 
these revisions. 

Example: 

The state-level agency develops a demonstration packet consisting of the awareness video from step 
1 above, an hour-long documentary film demonstrating the use of the proposed criteria and existing 
evaluation plan in a selected area vocational school. The agency also develops a set of hands-on, 
small-group activities; the awareness video, film and attendant activities are used for dennonstration 
presentations in every program provider site in Iniibna. Local governing boards, administrators, 
teachers, and students participate in the presentation. Based on written evaluation, the demonstration 
activities are revised to encourage more active participant involvement. Each local i>rogram provid- 
er site receives a copy of the revised demonstration packet and a brief description of die reasons 
for the changes. 

J. Te^ the proposed criteria. 

Prior to inatallAdon of the criteria, the various audiences need further evidence that the criteria and 
evaluation pUu will work in actual practice. To take this step, the state-level agency should develop 
appropriate materials and field test activities designed to help the audiences develop an understand- 
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ing of how the criteria and evaluation plan work in actual practice. Using these materials field test 
activities, the agency should 

a. select the field test sites from \ representative pool of advocates, neutrals, and detractors; 

b. determine the nature and extent of field-testing required to insure effective future commitment; 

c. conduct and evaluate the field test activities; 

d. revise the criteria and/or the evaluation plan based on the results of field test activities; and 

e. inform the awareness, demonstration, and field test audiences about the revisions and the 
reasons for thrse revisions. 

If the field tests failed to maintain the advocates, coopc the neutrals, and sway some of the "detrac- 
tors," it is critical to further field test the revised criteria and/or evaluation p!an. 

Exeuiyyle: 

Based on the results of the demonstration activities, the state-level agency selects 12 local program 
provider sites for field test activities: 4 apparent advocates, 4 apparent neutrals, and 4 a|^)cfent detrac- 
tors. The agency selects and assigns a full-time field test manager to each field site. The field test 
manager works for a year with local staff and agency staff to develop field test materials and activi- 
ties. During and at the end of the second year of field-testing, materials and activities arc revised 
as appropriate. Suminative results of the field test are provided to the awareness and demonstration 
audiences. 

4. Install the criteria. 

Prior to accepting the criteria as part of their daily way of life, the various audiences need clear 
evidence that the widespread application of the criteria will produce results which are proper, useful, 
feasible, and accurate. Most state-level agencies recognize the need for awareness, demonstration, 
and field test activities prior to installation of innovations at the local level. However, too often, 
they view the installation step as the final step in implementing/adopting the innovation. Research 
indicates that the installation of an innovation requires a level of statewide effort and suppon that 
is far in excess of the effort and support expended in the awareness, demonstration, and field test 
activities. Without that level of effort, the innovation is more likely to join that vast ^kxA of unused 
"good ideas" that never change actual practice in the schools. 

"^o take this step, the state-level agency should develop appropriate materials and installation proce- 
(iares designed to accomplish the installation of the criteria. Using these materials and procedures, 
the agency should 

a. conduct and evaluate installation procedures in those field test sites where field lest results 
indicate a relatively high probability of successfiil installation; 

b. revise the criteria and/or evaluation plan based on the results of these initial installation 
procedures; 

c. inform the various audiences about the revisions and the reasons for those revisions; 

d. continue to broaden the installation audience and repeat the revision and installation activi- 
ties detailed in 4b and 4c above; and 

e. inform the various audiences about the final revisions and the reasons for these revisions. 

Example: 

Working with local program provider sites and the 12 field test managers, the state-level agency 
develops a three-year installation plan. Using the revised field test materials and activities and the 
cadre of 12 field test managers, the state-level agency carries out the installation plan, revising the 
materials and activities as appropriate. Periodic newsletters report the progress of installation activi- 
ties to the awareness, demonstration, field test, and installation audiences. Newsletters contain such 
items as: examples of criterion application, smdent case studies, and information on successful adap- 
tations. 
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5. In&titiaionm^je the criteria, 

Contraiy to popular opinion, the institutionalization decision rests not with the state, but rather with 
the individual student, teacher, program site, and the governing body. The state can, of course, re- 
quire comfrfiance with the criteria and require evidence of that compliance. However, whether or 
not the criteria become an integral part of the program depends on whether or not the criteria are 
seen as proper, useful, feasible, and accurate by those most directly impacted by these criteria. 
To take this step, the state should 

a. encourage and support local ad^xations of the criteria and/or evaluation plan where these 
adaptations seem clearly in the best interests of all concerned; 

b. evaluate the results of these ad^Hations; 

c. incorporate successful adaptations; and 

d. continue this adiq)tation/evaluation/incorporation process for the duration. 
Example: 

During and after the three-year installation process, the evaluation plan is revised, and a **final" evalu- 
ation plan is presented at a statewide meeting of vocational educators. The 12 field test managers 
continue as resource people for local program provider sites to assist in the local adaptation process 
and to ensure that these adaptati(His are iqjpropriate. The newsletter continues to report the results 
of evaluation activities. Program provider sites with exemplary evaluations receive student scholar- 
ship awards, teacher incentive grants, and additional noncategorical vocational funding. Less exem- 
plary but conforming program providers are offered financial mcentives to continue to improve their 
evaluation activities. Nonconforming program providers are required to file a detailed plan for cor- 
rection and are subject to on-site state monitoring of correction plan implementation. Nonconform- 
ing program providers which fail to submit an acceptable plan for conducting their evaluation abrogate 
their right to locally directed program evaluation and are subject to full evaluation by the state-level 
agency. (Note: The sample: sanctions for nonconforming program providers do not include a reduc- 
tion in fundim;. Such reductions usually result in a lowering of program quality which, in turn, results 
in lowering tiie quality of education for youth.) 

6. Develop an evaluation foUow-up system. 

Whether the evaluation process be formative or sununative, the ultimate purposes of vocational and 
technical education program evaluation efforts are to determine the extent to which the goals of the 
program have been achieved, to improve the delivery of the program to the participants, and to im- 
prove the performance of \ht program participants. Acceptance of this premise assumes that the evalu- 
ation process continues beyond t^e completion and submission of the evaluation report. In this additional 
phase, program administrators use the evaluation results to gr'-^e their efforts to improve vocational 
and technical education programs by 

a. reviewing the evaluation results to identify deficiencies or problem areas; 

b. selecting areas for improvement; 

c. developing a foUow-up plan and schedule; 

d. implementing the follow-up plan; 

e. providing periodic progress reports to the staff, community, and state agency; and 

f. using the information and experiences gained through each follow-up plan in the develop- 
ment of other follow-up activities and subsequent program development efforts. 

Example: 

During and after the installation of the evaluation process, the structure of the follow-up plan is re- 
vised, and program provider administrators discuss their oqieriences with foUow-up plans at a statewide 
meeting of vocational educators. The 12 field test managers continue to work with local program 
provider sites in the design and implementation of follow-up activities. The newsletter continues to 
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report the results of cvaluation activities. Program providers compare the evaluation results with 
goals in the long-range r'an, analyze the evaluation results, and divide the deficiency or problem 
areas into three groups - correctable with minimal resources or effort, correctable with significant 
reallocated resources or effort, and only correctable with large resources or major restrucniring. Pro- 
gram providers develop a follow-up plan with a schedule for addrrssing the selected areas for im- 
provement. Program provider sites with exemplary follow-up activities receive financial support to 
implement meritorious reconunendations in the follow-up plan. Program providers with less exem- 
plary but conforming follow-up plans are offered financial incentives to design and implement their 
evaluation follow-up activities. If a program provider fails to develop follow-up plans, the program 
provider may be subject to state monitoring of efforts undertaken to correct deficiencies and possible 
fiill evaluation by the state-level agency. 

With systematic and persistent attention to the implementation and institutionalization of the evaluation 
process, the state may look forward to forming cooperative relationships with local school districts, 
other program provi Jers, and various stakeholders toward using evaluation to improve the quality 
of vocational/techn/cal education for the state's citizens and economy. 

Program Census Informatfon 

The development and maintenance of a data base about vocational and technical education at the 
local level is a critical element in the continuous process of evaluating the programs. Working with 
state agency and service provider personnel, the commission should develop a plan for gathering 
?m analyzing program census information. Some might be gathered for all programs, and others 
might be gathered on a sample basis. By using a cooperative approach in identifying the data ele- 
ments, the level of cooperation and validity of data should be enhanced. 

The following items are illustrative of those that might be included in a state-level data base that 
in actuality is a collation of information from the program provider level integrated with statewide 
demographic and labor market data. Examples of items include. 

• short- and long-term local and state job market projections 

• types of programs - classification scheme 

• student participation by program 

• student completers by program 

• number of students employed full/part-time 

• number of students employed in training-related field 

• student tenure on-the-job 

• smdent salaries and promotions 

• employer satisfacticm on an individual employee basis 

• student satisfaction with the program 

• number of program completers enrolled in postsecondary institutions 

Summary of Implementation Plan 

The following six sequential steps for the proposed implementation plan for program providers have 
been summarized from the previous discussion to provide a summary presentation of the process. 

1 . 'reate an awareness of the proposed criteria 

a. Idtintify audience 

b. Determine necessary level of awareness 

c. Conduct and evaluate orientation and training 

d. Revise criteria and/or evaluation plan 

e. Inform audiences 
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2. Demonstrate application of criteria 

a. Identify audiences 

b. Determine extent of demonstration required 

c. Conduct and evaluate demonstration activities 

d. Revise criteria and/or evaluation plan 

e. Inform awareness and demonstration audiences 

3. Test the proposed criteria 

a. Select field test sites 

b. Determine nature and extent of field-testing 

c. Conduct and evaluate field test activities 

d. Revise criteria and/or evaluation plan 

e. Infonn awareness » demonstration » and field test audiences 

4. Install the criteria 

a. Conduct and evaluate installation procedures 

b. Revise criteria and/or evaluation plan 

c. Inform various audiences 

d. Broaden installation audiences and repeat revision of installation activities 

e. Inform various audiences 

5. Institutionalize the criteria 

a. Encourage and support local adaptations 

b. Evaluate results of adaptations 

c. Incorpoiate successful adaptations 

d. Continue adaptation/evaluation/incorporation process 

6. Develop an evaluation follow-up system 

a. Review the evaluation results to identify deficiencies or problem areas 

b. Select areas for improvement 

c. Develop a follow-up plan and schedule 

d. Implement the follow-up plan 

e. Provide periodic progress reports to the staff, community, and state agency 

f Use the information and experiences gained through each foUow-up plan in the develop- 
ment of other follow-up activities and subsequent program development efforts* 
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PARTY 

Review of Current State Agency 
Evaluation Procedures 

Indiana's administrative structure for vocational and technical education is different from that found 
in other states. CVTE functions as an oversight and leadership agency for vocational and technical 
education in Indiana, while other agencies have a direct relationship to program providers at the lo- 
cal level. One aspect of the project to develop an evaluation system for Indiana vocational and tech- 
nical education was a review and summary of the current evaluation systems being used in Indiana. 
Special attention was given to the manner in which these systems related to state and federal regula- 
tions and statutes. The current evaluation systems of the following state agencies were reviewed: 

Indiana Conunission for Higher Education (CHE) 
Indiana Department of Education (DOE) 

Indiana Department of Employment and Training Services (DETS) 

Each of these agencies is actively involved in vocational and technical education, but they have widely 
different functions. Great diversity exists in their perceived roles in administering and evaluating 
vocational and technical education programs. 

The analysis of the current status of evaluation in the three state*level agencies was conducted in 
four phases. Phase I involved the review and analysis of state and federal statutes. The purpose was 
to gather information related to compliance and evaluation requirements. (The findings are shown 
in a matrix in accompanying Research Report No. 1.) 

In phase n, documents from the three state agencies were analyzed. CVTE provided relevant docu- 
ments from the commission's files. These documents were examined to secure information on cur- 
rent evaluation practices. Notations were made concerning compliance and evaluation requirements. 
In addition, an annotated bibliography was prepared for each reviewed document. (See accompany- 
ing Research Report No. 1.) 

Phase in involved interviews with the state-level personnel. The intent was to clarify /verify infor- 
mation, correct conflicting information, and provide omitted information. Lists of minimal criteria 
for Carl D. Perkins Vocational Education Act, Job Training Partnership Act, findings, and docu- 
ment reviews are contained in Research Report No. 1. 

Compliance and evaiu:»tion requirements were placed into a matrix in phase IV. The results were 
compared with the legislative con^Uance matrix referred to above. A discrepancy analysis was con- 
ducted to determine the degree to which Indiana was meeting the various state and federal mandates. 
Results of this analysis are discussed below. 

For CHE, vocational and technical education is a relatively small part of the total operation of 
higher education in Indiana* Even though CHE does not view itself as a direct service provider, the 
agency is the state-level agency for vocaticmal and :<xhnical education programs conducted by the 
Indiana Commission on Vocational and Technical Education, Vincennes University, Ball State Univer- 
sity, Indiana State University, Indiana University, Purdue University, and the Southern Indiana Univer- 
sities. CHE does not assume an active role in managing or monitoring programs and does not appear 
to have a comprehensive evaluation program for vocational and technical education. 
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DOE is the state-level agency for vocational and technical education programs conducted by the 
3Q2 local school corporations and 49 area vocational districts in Indiana. VocatiCMial and technical 
education was treated separately in DOE for many years, but current efforts are directed toward 
integrating vocational and technical education into the overall school program. In the local school 
corporations, planning, evaluation, and delivery of vocational and technical education is not treated 
separately. Evaluaticm of vocational and technical education programs appean to take place within 
the context of dw evaluation of regular education programs. With the area vocational districts, the 
situati(m is different because the entities ore essentially separate single-purpose bodies. 

DETS is the Jtaie-level agency for programs funded through the U.S. DqMitment of Labor. Leader- 
ship and technical assistance are major responsibilities of the agency. In contrast to CHE and DOE, 
the community-based organizations that serve as the local-level program providers under DETS have 
limited functions. Program evaluation requirements from DETS appear to be more oriented toward 
data reporting and compliance with regulations rather than toward concerns about the quality of the 
program. This state agency's activities are greatly influenced by the federal legislation. 

In view of die variations in tradition and perceived mission among the three state-level agencies, 
CVTE*s challenge is to develop effective working relationships with each agency and to work to- 
ward development of an integrated cooperative and interactive support system to deliver vocational 
and technical education in Indiana. Currently, CVTE appears to have major planning and evaluation 
responsibilities for vocational and technical education, but authority for conducting the programs 
is spread among a variety of state and local agencies. The operational challenge is for CVTE to in- 
crease the ctnnmitment and involvement of CHE, DOE, and DETS in cooperative planning and evalu- 
ation to improve vocational and technical education in Indiana. 

The contention of CHE leadership was that typical program concerns such as administration and 
curricular matters were instimtional responsibilities rather than interests of CHE. The agency per- 
ceived its role to be more related to serving the legislature and the governor in strategic planning 
and policy analysis than to gathering and analyzing data or monitoring and evaluating programs in 
institutions. As with DOE, CHE appears to contend that vocational and technical education should 
be treated in the same manner as oth^r programs in its constituent institutions. However, CHE does 
not appear to be assuming an active role in evaluating or monitoring higher education programs. 

For DOE, the discrepancy analysis indicated that Indiana was meeting most of the compliance 
requirements of the federal vocational and technical education programs under the Perkins Act. The 
assessment was that DOE had gathered sufficient information to pass a compliance audit. 

DETS {4>peared to be meeting or exceeding the external evaluation requirements from the federal 
level. One reason may be related to the prescriptiveness of the program outcome and evaluation criteria 
under the federal Job Training Partnership Act (JTPA); agencies have virtually no flexibility. Re- 
quirements are clear and concise. The unanswered issues are the extent to which the data gathering 
and evaluation systems used for the JTPA programs should remain intact under a more comprehen- 
sive system and the degree to which the current JTPA systems should be expanded to enhance the 
contribution that DETS makes to the development of a broad system of vocational and technical edu- 
cation and employment and training services for Indiana. 

The findings from the review and analysis of the current evaluation activities and the existing ad- 
ministrative structure for vocati(Hial and technical education in Indiana suggest that any proposed 
evaluation system should use a tiered approach rather than a smgle set of criteria. CVTE has a plan- 
ning and leadership misskm; the agencies (CHE, DOE, and DETS) have an administrative and monitor- 
ing function; and the actual instruction or training is provided by the various local program providers. 
Statewide (banning, statewide needs assessment, program development, and accountability are the 
major interests of CVTE. Program development, technical assistance, and compliance are major con- 
cerns of the state agencies. Program relevance and effectiveness are interests of die program providers. 
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PART VI 
Research on Evaluation of 
Vocational and Technical Education 

The task of reviewing the literature on evaluation of vocational and technical education was contract- 
ed to the Evaluation Research Center at Western Michigan University. (This group's complete re- 
port is contained in accompanying research reports nos. 2 and 2a.) In this summary, the results of 
the center's work have been organized around the following broad headings: 

Evaluation Framework 

General Recommendations 

Characteristics Reconmiended by Authorities 

Evaluation Framework 

Stufflebeam's CIPP model was used as the organizing construct in the discussion of an evaluation 
framework. Elements in the model are context, input, process, and product. Rather than being a 
model that only provides summative evaluation information, the CIPP model also provides forma- 
tive evaluation data that may be used to guide planning decisions, change program strategy choices, 
and gukle the implementation of an activity. Traditional summative infomiation also is provided through 
records of the process and the extent to which goals are attained (Madaus, Scriven, and Stufflebeam, 
1983). 

Given the data burden and tlie staff time requirements associated with traditional evaluation models 
used by the regional accrediting associations and professional organizations, the increased attention 
being given to outcomes evaluation is a natural evolution. Campbell and Panzano (1985) recommended 
that a set of indicators be used to assess the quality of a program; these indicators would then be 
used to measure the outcomes and assess the efficacy of vocational and technical education programs. 
Their reconmiendations are consistent with earlier work of Darcy (1979) that stressed a need for 
data on outcomes; this information was then to be used as the basis for an analysis, interpretation, 
and evaluation of particular programs. 

General Recommendations 

The appropriateness of using traditional evaluation models has been questioned by reports from the 
vocational and technical education community. The focus on quantitative inputs has been viewed 
as incompatible with the complexity of vocational and technical education delivery systems (Com- 
tnittae on Vocational Education Research and Development, 1976). Later work has emphasized the 
need for the use of critical evaluation as a mode of inquiry to recognize environmental constraints 
and interpret the impact of various conditions (Farley et al., 1985). 

From an analysis of federal legislation, Grasso ( 1979) proposed an evaluation model that had three 
components: process, product/jutcome, and product/impact. Process includes program rationale, 
program content, program deliverers, program recipients, and program environment. Product/out- 
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come refers to career and labor market infonnation on program completers and leavers. Student com- 
petencies and proficiency are the focal points of product/impact data. 

Brannon's (1983) analysis of the Perkins Act stressed the importance of evaluation being related 
to i»ogram efEiBctiveness. He suggested the following measures of program effectiveness: occupa- 
tions that relate to the labor nuurk^ needs of the state, skill levels aixl competencies that will meet 
the hiring needs of ennployers, and the effects of these on handicapped persons seeking employment. 
CXher program evaluation areas noted by Brannon included the relationship between vocational and 
technical education and economic develofment, the importanc; of addressing the needs of special 
populations, the need to provide an adequate level of student services, and the need to decrease sex 
role stereotyping. 

Further direction for efforts to evaluate vocational and technical education can be found in the 
results of a survey of state direcnors of vocational and technical eduv jtion by Wentling and Barnard 
(1982). In response to a request that they establish rankings of eight possible indicators of program 
qualitv, the state directors' responses in rank order were as follows: employer feedback, placement 
level, quality curriculum materials, job satisfaction, and a 'bur-way tie of instructor performance, 
student test performance, condition of equipment, and scSool staff models. Of the eight indicators, 
four might be classified as effectiveness, outcome, or product criteria; fjvo as process criteria, and 
two as input criteria. 

Even though a body of literature on program evaluation exists and many evaluation procedures appear 
to be applicable to education, questions have been raised about the extent to which these established 
evaluation concepts could be used to develop a usable evaluation system for vocational and technical 
education (Gourley, 1989). Issues in developing an evaluation system for vocational and technical edu- 
cation were noted by Leske and Pbrsico (1984); their review of the literature suggested a lack of agree- 
ment about the goals of vocational and technical education and the existence of measurement problems 
for a dynamic enterprise. In a survey of local vocational directors in RUnois, differences in emphasis 
between secondary and postsecondary education and variations in employer expectations were noted 
fay Wentling and Barnard (1982). Their findings supported the need to fbcus on smdent outcomes in 
vocational and technical education. However, problems related to inadequacies of data bases for as- 
sessing program effectiveness were noted in a Pennsylvania smdy (Buttram et al., 1987). 

State-level evaluators supported the movement toward outcome or effectiveness evaluation in their 
responses to telephone interviews (Gourley, 1989). Outcomes or effectiveness evaluation approaches 
require that there be some agreement about desired outcomes and levels cX attainment; Mitzel (1982) 
contended that no single criterion is likely to be satisfactory for evaliiiating vocational and technical 
education programs and that no national consensus exists on a limited set of outcomes that can serve 
as evaluation criteria. However, this issue can be resolved by setting goals for vocational and techni- 
cal education within the environmental context of each service area (Silberman, 1983). 

Movement toward an outcome or effectiveness emphasis has significant implications for data re- 
quirements. Starr (1986) proposed that data be collected in five groupings: employment context of 
the service area, vocational and technical education process, outputs, outcomes, and benefits. For 
the first three, consensus on data elements should be achieved quickly; for the fourth, some debate 
might be raised about appropriate outcome data; but for the fifth, issues will include acceptable indi- 
cators of benefits and the amount of time that must pass before the benefits can be determined. 

Characteristics Recommended by Authorities 

The principal concerns of authorities in the design of an evaluation system for vocational and techni- 
cal education appear to be activities that are related to establishing, prioritizing, and assessing the 
attainment of goals. McCaslin and Lave (1976) identified five major tasks in the process: learner 
outcomes, instrumentation, sampling strategies, relationships between actual and desired achieve- 
ment levels, and identification of priorities. 
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Several problem areas related to the evaluation of vocational and technical education were identi* 
fied by Stevenson (1979); they can be grouped into the following categories: 

• expansion and systematization of evaluation 

• follow-through on evaluation 

• seivices for special populations 

• evaluation of the state delivery system 

• cooperation with and follow-up data from employers 

• use of evaluation information 

A national study indicated thn process and outcomes evaluation programs can help vocational and 
technical education address current evaluation problems (National Commission, 1984). This nation- 
al commission identified such areas as agreement on goals, access, and equity; curricula and teach- 
ing practices; standards and measures of accountability; articulation between educational levels; criteria 
for leadership; partnerships with business, labor, and community interests; and evidence of field- 
based learnings. 

Summary 

The content of the various reports summarized from the review of the literature suggests the follow- 
ing observations: 

• Federal legislation is a major influence behind the interest in the evaluation of vocational and 
technical education. 

• Literature on evaluation of vocational and technical education and statutory requirements ap- 
pear to focus on evaluation efforts as top down from the state level rather than bottom up 
from the site at which programs are delivered. 

• Current emphasis in evaluation of vocational and technical education appears to be more on 
program effectiveness than on program inputs or the process of instruction. 
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PART MI 
Practices in Other States 

CVTE staff expressed an interest in certain practices used in specific states, and members of the 
National Advisory Committee indicated that information about practices in other specified states should 
be of value to the study. Following these directions, specific evaluation procedures were reviewed 
and analyzed in the following states - Colorado, Florida, Georgia, Illinois, North Carolina, and 
Wisconsin. Practices varied among the states, but several common elements have relevance to Indi- 
ana. Summary tables on the evaluation focus in the sample states and a more complete discussion 
of the practices in each state arc presented in accompanying Research Report No. 2. 

Colorado. 

The evaluation program focuses on four separate activity areas - local review, program approval, 
compliance, and state services. The state's plan is referred to as the **Q Process," which involves 
a self-review followed by a validation process. Reports from the state indicated a heavy emphasis 
on process with limited attention to program outcomes. The process included three phases - re- 
view, validation, and technical assistance. Criteria checklists are used to evaluate 

• the instructional program 

• program facilities 

• business and community resources 

• vocational guidance and counseling 

• student placement and follow-up 
•. administrative support 

Following the self-review phase, an improvement plan is developed by the program faculty. Results 
of the self-review and the improvement plan are validated. The next phase is to prepare a local pro- 
gram action plan. The last phase would involve state technical assistance to continue the program 
improvement process. 

Florida. 

The primary purpose of the Florida evaluation program is to provide the state with accurate follow- 
up information on former secondary and postsecondaiy program participants. Then, such informa- 
tion is to be used in evaluating publicly financed training programs. Data for each student are main- 
tained at the state level through the use of the participant's Social Security number. 

Through Florida's comprehensive management information system, information is available on age, 
race, sex, programs taken during schooling, and current occupaticms. Actual jobs are matched with 
those in the Dictionary of Occupational Titles. These data permit an analysis of earnings and labor 
market trends for use in career guidance and counseling of students. 

Georgia. 

Considerable attention has been given to a five-year process of field testing the evaluation process 
in various sites around die state. The process involves 55 specific course content guides with selected 
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occupational skills. Vocational evaluation criteria have become infused into guides that local school 
di^cts use with all program participants. 

This state's evaluation policy was an outgrowth of the state's long*range plan. The transition plan 
involves a three-stage computer-assisted evaluation program. Stages are basic indicator assessment, 
in-depth analysis, and program improvement planning. Basic indicators are placement, enrollment, 
employer satisfaction, student satisfaction, student performance, cost containment, and labor market 
justification. With this information, local personnel pose a series of questions to the computerized 
••expert system*" 

Program improvement planning utilizes the data and analysis from the first two stages. These in- 
terpretations contribute to the development of program improvement plan. Steps to be taken to im* 
prove the programs are noted, and the requests are submitted to the state for additional funds. 

North Carolu!a. 

A joint state and local planning effort is used to maximize the contribution made by agencies, groups, 
or individuals interested in vocational and technical education. Advisory committees are used to sur- 
vey community job needs, advise on course content, review student placement data, make recom- 
mendations on facilities and equipment, identify potential teachers and community resources, review 
evaluation data, and advise on local plan development. 

Wisconsin. 

Rather than having a single state agency responsible for vocational and technical education, Wiscon- 
sin has one agency for secondary-level vocational and technical education and a separate postsecon- 
daiy vocaticHnal and technical education system with an independent state board. Postseomdary programs 
and services are provided in 16 regional districts. Incentive grants are used to encourage the introduction 
of programs for special populations and upgrade existing programs. Programs are available for sec- 
ondary, postsecondary, and adult participants. 

For its evaluation system, Wisconsin focuses on five outcomes to assess results: employment of 
students after leaving school, secondary graduates in postsecondary education, satisfaction of stu- 
dents and employers with training, differences in satisfaction levels among types of participants, and 
differences between regular and special needs students. Program components are assessed on 17 criteria 
that include the existence of a i^losophy, needs assessment, comprehensive curriculum, attention 
to equity, programs for emerging occupations, sufficient equipment and facilities, existence of an 
advisory conunittee and student organization, follow*up studies with graduates and employers, and 
implementation of the recommendations from prior evaluations. 

Evaluation procedures varied among the selected states; the conunon element appeared to be the 
development of a state-level data base about local programs and/or program participants. Rather than 
relying on input criteria to indicate the existence of essential preconditions, evaluation in the selected 
states tended to focus on outcome, product, or effectiveness criteria. 
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APPENDIX 
Effectiveness and Process Criteria 

for Evaluating Indiana 
Vocational and Technical Education 

The following effectiveness and process criteria for evaluating vocational and technical education 
in Indiana are presented in three groups - local program provider, %wt agency, and commission. 
One or more process criteria are presented for each effectiveness criterion in the appendix. Process 
criteria should be viewed as illustrative; the assumption is that deletions, additions, and adaptations 
will be made when ap|»t>priate. Program administrators would use process criteria to indicate the 
extent to which each of the effectiveLjsi criteria have been attained in the local setting. 

Some effectiveness criteria appear in all three sets, but the emphasis varies depending on the agen- 
cy's specific responsibilities. The conmiission has greater leadership and planning responsibilities. 
The state agencies have greater monitoring and technical assistance responsibilities, and program 
providers have greater delivery and evaluation responsibilities. 

In this report, the term "conmussion's long-range plan" refers to Education and Jobs for Indiana's 
Future, the long-range plan for a comprehensive vocational program in Indiana, as approved by the 
commission on July 14, 1988. This plan, agency plans, and local plans are different from those re- 
quired by the U.S. Department of Education to participate in federal programs. 

Effectiveness and Process Evaluation Criteria 
for Program Providers 

The following list of effectiveness and process criteria for the program providers was developed as 
a response to task 4 in the project contract. Program providers include the state's 302 school corpora- 
tions, 49 area vocational districts, 7 public postsecondaiy instimtions. independent colleges and univer- 
sities, private proprietary schools, community-based organizations, and other agencies. Criteria for 
evaluation at this level have been developed from the material submitted by the various contractors. 
In this document, the criteria have been divided into the following areas: 

1.0 Structure 

2.0 Pbuming 

3.0 Diagnostic Assessment 

4.0 Program Design and Delivery 

5.0 Participant Services 

6.0 Partncrdiips 

7.0 Fadllties and Equipment 

8.0 Participant Outcomes 

9.0 Program Accountability 

10.0 Fiscal Accountability 

11.0 Cost-Effectiveness 
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These evaluation criteria are designed to fiEK;ilitate tracking the progress of program delivery sites 
in im|riementing those strategies outlined in the state and local long-range plans and to perroit refine- 
ment of specific evaluatim criteria as strat^ies axe impleinented. The process criteria may be used 
by each program delivery site in assessing the extent to which an effectiveness criterion has been 
attained and in determining responses to identified problems. 

1.0 Structure 

Effectiveness Criterion 

1.1 Philosophy and Organization - Program provider's philosophy statement and adminis- 
trative organization are relevant to the immediate service area, reflect current conditions, 
and arc consistent with and supportive of current programs and documents from the com- 
mission and the relevant state agency. 

Process Criteria 

1.1.1 Philosophy and Goals - Program provider has a statement of philosophy with 
related goal statements that express the principles and concepts adhered to in plan- 
ning and providing programs. 

1.1.2 Organizational Arrangements - Program provider has a governance and adminis- 
trative structure tfiat promotes citizen access and involvement in planning, providing, 
and evaluating programs. 

1.1.2a Defined Roles - Clearly states the roles, expectations, and levels of 
authority for vocational administrators. 

1.1.2b Commitment - Indicates a continuing commitment to providing voca- 
tional education programs. 

1.1.3 Legal Compliance - Program provider admit jstrators assume responsibility for 
ensuring that programs meet stamtory and regulatory requirements. 

1.1.4 System Consistency - Program provider ensures that program philosophy , goals , 
and administrative organization are consistent with and supportive of the mission 
statements of the commission ^id relevant state agency. 

2.0 Phuming 

Effectiveness Criterion 

2.1 Local Comprehensive Plan - Program provider's current and proposed programs are 
consi«(tent with and supportive of the program provider's comprehensive long-range plan 
and long-range plans of the commission and the relevant state agency. 

Process Criteria 

2.1.1 Planning System - Program provider utilizes a comprehensive planning system 
that provides opportunities for participation and interaction among relevant par- 
ties, relies upon and utilizes basic data, includes a comprehensive evaluation sys- 
tem, and utilizes information about current and potential programs, program 
participants, and employers. 

2.1.2 Plan Consistency - Program provider's comprehensive long-range flan is con- 
sistent with and supportive of the mission statements and long-range plans of the 
commission and relevant state agency. 

2.1.3 Stakeholder Involvement — The various parties affected by program changes are 
effectively involved in the planning process. 
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1 2. L3a Current and Potential Employers - Involves an employer advisory group 

at various stages of program planning and evaluation. 

B 2. L3b Staff - Invdves staff at various stages of program planning, implemen- 

■ tation, and evaluation. 

2.1.3c Program Participants — Involves program participants at various stages 
I of program planning, implementation, and evaluation. 

2.1.4 Linkages - Effective c(xnmunication channels exist between the program pro- 

■ vider and the immediate service area and between the program provider and pro- 
P gram providers in other agencies and institutions* 

^ 2,1.5 Environmental Scanning - Program provider has an ongoing system for collect* 

■ ing, updating, analyzing, and integrating relevant information into the planning 

process* 

H 2.L5a World-of-Work/Labor Market Data - Uses current and projected in- 

p formation about the general world-of*work and the related specific em- 

ployment needs in program planning* 

■ 2J.5b Environmental Data - Uses current and projected employment, eco- 
p nomic, and demographic data about the inmiediate and extended ser- 
vice areas in projecting specific employment needs. 

I 2*1 .Sc Employee Charaaeristics — Uses world-of-work leadership profiles in 

W . the projection of desired employee characteristics* 

2.L5d Demographic Projections — Studies and analyzes projected population 

■ shifts for the immediate and extended service areas in temis of their ef- 
' feet on the potential pool of program *)articipants. 

^ 2* L5e Potential Staff - Gathers and analyzes data about the age and experience 

I profiles of current staff and projections about the potential pool of new 

staff from traditional and nontraditional sources in terms of potential im- 
^ pact on staff availability. 

P 2.1.5f Funding Levels - Maintains and uses current information on anticipat- 

ed changes in state and federal funding levels and/or legislative reauthori- 

■ zations in program planning. 

2.1.6 Formative and Summative Evaluation - Program provider adopts and imp \;ments 
an evaluation system, and the resulting data are collected, updated, analyzed, and 

■ integrated mto the planning process. 

2.1.da Program Improvement - Uses results of formative evaluations to guide 
M program in4)rovement during the course of a program. 

P 2* 1.6b Program Planning and Revision - Uses results of summative evalua- 

tion for accountability and reporting to revise current programs and plan 

■ new initiatives. 

2*1.7 Program Participants - Program provider has an organized process for gather- 
^ ing infomiation about program participants, and these data are collected, updat- 

■ ed, analyzed, and integrated into the planning process. 

2«1.7a Skills, Abilities, and Interests - Collects, updates, analyzes, and inte* 
m grates information in a systematic manner regarding the skills and oc* 

■ cupational and career preferences into the planning process. 



2.1.7b Demographic Information - Collects, updates, analyzes, and integrates 
demogr^>hic information about program participants into the planning 
process. Data items include gender, limitcd*English*speaking, special 
needs, race, and ethnicity. 

2.1.7c Curricula - Maintains a data base of curricular options that addresses 
course goals and program participant assessment data requirements. 

2.1.7d Materials - Bases current instructional materials on course goals and 
program participant assessment data, and provides for the periodic re- 
view and updating of these materials. 

2.1.8 Program - Program provider has an organized process for gathering informa- 
tion about the available programs and staff, and the data are updated, analyzed, 
and integrated into the planning process. 

2.1.8a Staff - Collects and analyzes information about the professional and 
technical qualifications of the stafT analyzed on a regular basis and inte- 
grates the data into the planning process. 

2.1. 8b Program Adaptability - Has and uses procedures for adapting pro- 
grams and course content to the needs of program participants, 

2.1. 8c Facilities - Relates facility space and utilization to program content 
and program offerings. 

2.1.9 Business and Community Resources - Data about local business, industry, and 
conmiunity characteristics are collected, updated, analyzed, and integrated into 
the planning process. 

2.1.10 Annual Report - Program provider submits an annual report with an update 
of the strategic plan and a progress report on goal attainment to the appropriate 
agency* 

3.0 Diagnostic Assessment 

Effectiveness Criterion 

3.1 Diagnostic Assessment - Program participants and staff use diagnostic assessment in- 
formation in making decisions about program offerings and individual program selections. 

Process Criterion 

3.1.1 Assessment Process - Program provider has an organizes process for program 

participant assessment and diagnosis, and the data are updated, analyzed, and 

integrated into the planning process. 

# 

3.1.1a Knowledge, Skills, Aptitudes, and Application of Skills - For each 
program, uses standardized and nonstandardized tests to measure a per- 
son's cognitive, affective, and psychomotor domains and to assess knowl- 
edge, skills, aptitudes, and capacity to apply skills for entry-level 
program participants. 

3.1.1b Occupational Tests - When available, administers and uses results from 
standardized occupational or job-task criterion-referenced tests. 

3.1.1c Special Needs - Uses procedures to identify and diagnose the special 
needs of program participants. 
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iTogram Design and Delivery 

Effectiveness Criteria 

4.1 Program Design Decisions - t)ciivery systems accommodate the special needs of pro- 
gram participants and provide for exposure to and participation in actual work settings. 

Process Criteria 

4.1.1 Program Content and Delivery - In making decisions about program content and 
delivery, emphasis is placed on field work-setting experiences, program adapta- 
tion and responsiveness to participant needs, career awareness on the part of pro- 
gram participants, and staff development. 

4.1.2 Cooperative Education - Where applicable and feasible, the system for deliver- 
ing instruction provides cooperative education opportunities for program par- 
ticipants. 

4.1.3 Alternative Delivery Systems - The program delivery schedule inclu-es oppor- 
tunities for evening programs, exit and re-entry options, and observations and par- 
ticipation in applied settings. 

4.2 Curriculum Decisions — Program provider administrators use a systematic and compre- 
hensive approach in making curriculum decisions end use data related to the immediate 
service area, program participants, and the goals and functions of related agencies and 
institutions. 

Process Criteria 

4.2.1 Technology Education Curriculum - Program provider considers the technolo- 
gy education curriculum in making local curricular decisions. 

4.2.2 Development - Program provider and community resources are used in the de- 
velopment of the structure of the overall curriculum. 

4.2.3 Instructional Objectives - Written statements specify knowledge, competencies, 
and behaviors that program participants are to demonstrate under specified con- 
ditions. 

4.2.4 Competency-Based Instruction - Program learnings, experiences, and evalua- 
tions are structured to provide information that the program participants possess 
the desired competencies. 

4.2.5 Teaching Methods - Teaching methods and time on task vary depending on the 
needs of the program participants and the content. 

4.2.6 Participant Feedback - Instruaors seek ^d utilize participant evaluations and 
feedback in plaiming program delivery. 

4.2.7 Revision - Program participant assessment results are used to diagnose curricu- 
lum revision needs. 

4.2.8 Integration - Where applicable, the academic, general, basic skills, and voca- 
tional education course goals in the program provider institution are integrated 
into the total experiences of the program participant. 

4.2.9 Transfer of Learnings - Emphasis is placed on the use of skills and knowledge 
in multiple settings. 



43 



A-6 



4»2«10 Articulation — Program provider uses procedures for curriculum articulation with* 
in and among departments and institutions. 

4.3 Instructional Decisions * Program conqwnents arc designed to make participants aware 
of career opportunities* reinforce positive work attitudes, attain basic skills, develop em- 
ployability skills, and learn specific occupational skills and good safety practices. 

Process Criteria 

4*3.1 Awareness, Orientation, and Exploration - Potential and current program par- 
ticipants are provided opportunities to secure information related to career aware- 
ness, career orientation, and career exploration. 

4*3*2 Occupational Skills — Instructional programs focus on developing those occupa- 
tional skills related to the entry-level goals of the specific occupation. 

4.3.3 Safety - Programs emphasize the importance of safety procedures in both the 
school setting and the applied setting* 

4.3*4 Safety Instructions - Verbal and written on-the-job and in-school safety rules 
and procedures are provided to program participants and enforced. 

4.4 Staff Development - Program provider takes steps to ensure that the staff has and will 
continue to develop the qualifications and skills to provide the selected programs. 

Process Criteria 

4.4.1 Personnel Recruitment - Job opening information is unbiased, nonrestrictive, 
and designed to attract a talented group of employees. 

4.4.2 Personnel Preparation - Personnel possess the specialized and general knowl- 
edge and relevant state license or certificate required for the job. 

4*4.3 Personnel Selection — Equal employment opportunity and affirmative action 
procedures are observed in selection of applicants. 

4.4*4 Staff Assessment - Staff development needs are developed after professional 
qualifications and skills of instructional and administrative staff are matched to 
program structure, program content, and program delivery* 

4.4.5 Staff Evaluation - An organized staff evaluation program provides an assess- 
ment of job performance and feedback regarding strengths and areas of concern. 

4*4.6 Inservice Programs - Staff development programs are being provided to respond 
to the identified needs of the staff . 

4*5 Funds - Required human and material resources are appropriately allocated within and 
among programs. 

Process Criteria 

4.5.1 Budget - Annual budgets are prepared to ensure that the funding level for each 
program is sufficient to provide the required human and nuiterials resources. 

4*5.2 Fiscal Analysis - Exper Jitures are analyzed to determine if funds were expended 
for the intended purpose and if modifications should be iriade in the budget for 
the next budget cycle. 
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■ 5.0 PartidiMuit Services 

Effecsiveness Criteria 

5.1 Program Access - Broad access is provided to the handicapped, disadvantaged, and 
W oiinorities. 

^ Process Criteria 

■ 5.1.1 Access and Equity - Provisions are being made for all interested persons to have 

access to programs and to allow for equity to all groups in program enrollment 
» and career guidance. 

• 5.1.2 Basic Enrollment Data - Annual course and program enrollment data are classi- 

fied and reported by race, sex, ethnicity, and other demographic characteristics. 

■ Data are updated, analyzed, and integrated into the planning process. 

5.U Special Needs Populations - Efforts are made to recruit program participants from 
M the total population including special needs groups and underrepresented groups. 

5.14 Special Population Data - Course and program enrollment data are categorized 
mm and analyzed by special needs and underrepresented groups and integrated into 

H the planning process. 

5.2 Program Ser -ces - Program provider selects and provides services to address the needs 
I of current and projected program participants. 

Process Criteria 

■ 5.2.1 Program Participant Special Needs - Efforts are made to assess the special needs 
P of program participants and to assess the feasibility of providing services to ad- 
dress those special needs. 

P 5.2.2 Special Needs Clients - Where applicable, special needs program participants 

have an individualized vocational education plan (IVEP). 

■ 5.2.3 Career Development - Program participant organizations are encouraged and sup- 

ported to facilitate leadership and career development, to supplement the program 
participants program, and to provide an extension and expansion of learning ac- 

■ tivities focused on career development. 

5.2.4 Counseling Services - Special staff and procedures are available to provide per- 
H sonal, academic, and career counseling services. 

5.2.5 Job Placement - Specific staff and procedures are available to assist pro- 
m gram ptirticipants in job placement while they are in school and after they leave 
H school. 

« 5.3 Program Breadth - Program participants have access to different choices within the pro- 

■ gram provider's cycle of offerings. 

Process Criteria 

■ 53.1 Program Participant Orientation — The orientation program provides participants 
m with information and program offerings. 

■ 5.3.2 Program Schedule - Programs are scheduled at a time convenient to the primary 
I target group. 
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5J3 Counseling Access - Program provider has a referral process for participants that 
provides participants with additional information and access to counselors upon 
request. 

ftrtnersliips 

^ectiveness Criteria 

6.1 Rutnership Programs - Program provider develops and maintains partnership programs 
between md among business, industry, labor, government, and education. 

Process Criteria: 

6.1.1 Business Partnerships — Program provider has a governance, administrative, and 
advisory structure that promotes business, industry, and labor involvement in plan- 
ning, providing, and evaluating programs. 

611.2 Stronger Relationships - Program provider has a plan for strengthenii^ existing 
partnerships with local business, industry, and labor groups. 

6.2 Technology Education Curriculum - Program provider involves business, industry, la- 
bor, government, and education in the design, development, and implementation of the 
technology education curriculum. 

Process Criteria 

6.2.1 Cooperative Activities - Program provider holds regular meetings with persons from 
business, industry, labor, government, and education to secure their assistance in the 
design, development, and implementation of the technology education curriculum. 

6.2.2 Implementation of Recommendations - Program provider keeps records of ad- 
visory conmiittee input and reconmiendations on implementation of the technical 
education curriculum. 

7.0 Facilities and Equipment 

Effectiveness Criterion 

7.1 Facilities and Equipment - The facilities and equipment meet acceptable standards of 
safety, adequacy, adaptability, accessibility, and public health. 

Process Criteria 

7.1.1 Equipment and Materials - Teaching and laboratory equipment, materials, and 
supplies are up-to-d£<[e, safe, and sufficient in quantity to support program offer- 
ings and services. 

7.1.2 Instructional Technology — Facilities can accommodate the technology required 
to conduct a current program. 

7.1.3 Accessibility — Structural features and layout of grounds and facilities cr>.ance 
participation by regular and special program participants. 

7.1.4 Adaptability — Design and layout of the grounds, buildings, and equipment are 
flexible and adaptable to meet changing demands for programs and services. 

7.1.5 Safety and Security - Design and ccHidition of grounds, facilities, and equipment 
are adequate to minimize the possibility of harm, injury, or loss. 

7.1.6 Health - Periodic inspections are made to ensure that no health hazards exist 
in the facilities. 
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8.0 Partkipant Outcomes 

Effectiveness Criteria 

8. 1 Demonstrated Competency - Program completers cin demonstrate the competencies that 
are the goals of the program and the expectations of employers. 

Process Criterion 

8.1.1 Assessment Procedures - Program provider has established systematic assess- 
ment procedures and expected levels of performance to determine if program par- 
ticipants have attained the desired level of knowledge and skills. Assessment 
information may be developed from the results of teacher-developed tests, teach- 
er observations, anecdotal records, activity logs, inventories, standardized tests, 
or other instrum^"'* .orovcd by program providers. Knowledge and skill areas 
to be considered to. cessment include \ht following: 

Career Planning - Personal awareness and development of interests, abilities, 
and skills; personal awareness of occupational opportunities; and skiUs and knowl- 
edge needed to prepare for and become established in an occupation. 

Work Habits - Skills and knowledge required to be a productive worker includ- 
ing knowledge of safety rules and procedures. 

Career Exploration — Exploration of occupations in simulated or applied settings. 

Employability Skills - Job readiness skills including resume writing or interview 
techniques. 

Specific Occupational Skills ~ Skills and duties related to a specific job. 

Basic Skills - Skills required to succeed in vocational education programs and 
on-the-job, i.e., communication, mathematics, and problem solving. 

Human Relations Skills - Skills and knowledge needed to work cooperatively 
with others. 

8.2 Job-Related Tasks - Program completers apply their knowledge and skills in securing 
and keeping a job. 

Process Criterion 

8.2.1 Data Base - Program provider has an organized process for gathering data to de- 
termine if program participants have attained the ability to apply knowledge and 
skills, and the data are updated, analyzed, and integrated into the planning process. 
Knowledge and skill areas to be considered for assessment include the following: 

Employability SkiUs - Time and efifort required to locate and secure employ- 
ment in training-related job. 

Basic SkilL'. - Application of skills in written and oral conununication and job- 
related proolem solving. 

Human Relations Skills - Inter()ersonal skills in working with peers, supervi- 
sors, or supervisees. 

Occupational Skills - Skill expectations for specific job. 

Work Habits - Self-discipUne, self-reliance, punctuality, and observance of job 
rules and procedures. 
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9.0 Program Accountability 

Effectiveness Criteria 

9.1 EnroUmeiit/Completioii Targets - Program participant enrollment levels and comple- 
tion rates are at, or exceed, the established standard. 

Process Criteria 

9.1.1 Data Base - Annual course and program enrollment data indicate the number 
of program participants who completed the training program in which they were 
enrolled. 

9.1.2 Data Analysis — Program provider analyzes enrollment and program comple- 
tion data to identify those factors associated with successful completion of the 
program. 

9.2 Related Employment - Program completers are employed in the jobs for which they 
were trained. 

Process Criterion 

9.2.1 Use of Data on Training-Related Employment - Data are gathered, updated, and 
analyzed concerning the number of program completers who are employed in jobs 
that might be classified as either training-specific or training-related. 

9.3 Gainful Employment - Program completers are employed in gainful occupations. 
Process Criterion 

9.3.1 Employment - Data are gathered, updated, and analyzed concerning the number 
of program completers who are placed in fiill- or pan-time employment. 

9.4 Salaries - Salaries paid to program completers working in a training-related job exceed 
those for persons who did not complete the training. 

Process Criterion 

9.4.1 Salary Survey - Program provider has an organized process for gathering data 
on salaries received by program completers, and the data are updated, analyzed, 
and integrated into the planning process. 

9.5 Job Stability - Program completers exhibit a specified level of employment stability. 
Process Criterion 

9.5.1 Follow-Up Data — Program provider gathers follow-up data concerning the length 
of tin;e that program completers spend in their job and compares this information 
V ith that of other pe sons in the same j( os. 

9.6 Employee Satisfaction - Program conq>leters indicate that they are satisfied with the con- 
tent, experiences, and outcomes of the training program, and have job satisfaction. 

Process Criterion 

9.6.1 Satisfaction with Program - Program provider has an organized process for 
gathering data on program completers' attitudes toward the program and cur- 
rent job, and the data are updated, analyzed, and integrated into the planning 
process. 

9.7 Employer Satisfaction - Employers are satisfied with the on-the-job performance of the 
persons en4>loyed from the program. 
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Process Criteria 

9.7.1 Employer Surveys - Program provider conducts periodic surveys and inter- 
views to secure evaluations of employer satisfaction with on*the-job perform- 
ance of individual employees and program completers as a group. 

9.7.2 Survey Analysis - Program provider coUects, continually updates, periodi- 
cally analyzes, and integrates survey data into the planning process* 

9.8 Postprogram Education - Program provider has follow*up information to determine 
the number of program completers who complete high school and/or pursue additional 
formal education. 

Process Criteria 

9.8.1 Postprogram FoUow-Up - Program provider has an organized process for 
gathering foilow*up data on continuing education and/or employment for pro* 
gram leavers and conq>letcrs, and the data are updated, analyzed, and integrated 
into the planning process. 

9.8.2 High School Completion - Annual course and program enrollment data indi- 
cate the number of program participants who completed high school gradua- 
tion requirements* 

9.8.3 Continued Education - Data are gathered, updated, and analyzed concerning 
the number of program completers who enroll in postsecondary institutions. 

10.0 Fiscal Accountability 

Effectiveness Criterion 

10.1 Fiscal Audits - Financial audit reports indicate that fiscal procedures are in compli- 
ance with applicable local, state, and fede/al rules. 

Process Criteria 

10.1.1 Periodic Audits - Fiscal accounting records are periodically reviewed by an 
independent auditor. 

10.1.2 Audit Reports - Audit reports inc icate minimal negative findings in the han- 
dling of funds. 

10.1.3 Sufficient Records - Fiscal accounting, staffing, and program participant 
records are in sufficient detail to determine the program cost per enrollee, 
program completer, and program placement. 

11.0 Cost-Effectiveiiess 

Effectiveness Criterion 

11.1 Cost-Effectiveness - Given the program objectives and compared to the cost of simi- 
lar programs, cost data per program enrollee, completer, and placement indicate that 
the program is cost^fTective. 

Process Criteria 

11.1.1 Data Base - Fiscal, participation, and perfomiance data are maintained for 
program enroUees, completers, and placements. 

11.1.2 Costs per Participant - On a periodic basis, program costs are compared and 
analyzed using similar data ^om comparable programs. 
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11.1.3 Attainment of Standards - Systematic procedures arc used to determine if 
program placements* on-the-job performance levels are meeting or exceeding 
local, state, and national performance standards. 

11.1.4 Cost-Benefit Analysis — Expenditure, performance, and completer data are 
compared to determine the effectiveness of the program. 

Effectiyeness and Process Evaluation 
Criteria for State Agencies 

The following list of effectiveness and process evaluation criteria for the state agencies has been 
developed as a response to task 4 in the project contract. The state-level administering agencies for 
vocational and technical educatim are the Conunission on Higher Education, Department of Em- 
ploymeikt and Training Services, and State Board of Education. The principal sources for these criteria 
were the commission's long-range plan, existing evaluation criteria, related statutes, and commis- 
sioned smdies. The criteria have been divided into nine broad areas: 

1.0 Structure 
2.0 Long-Range Plan 
3.0 Agency Leadership 
4.0 Fiscal AccoimtabUity 
5.0 Tedinical Assistance 
6.0 Program Evaluation 
7.0 Articulation 

8.0 Professional Personnel Development 
9.0 Partnerships 

These evaluation criteria are designed to facilitate tracking the progress of state agencies in im- 
pler .enting thoso strategies oudined in the commission's long-range plan and to permit refinement 
of specific evaluation criteria as strategies are implemented. The process criteria may be used by 
the agency in assessing the extent to which an effectiveness criterion has been attained and in deter- 
mining responses to identified problems. 

1.0 Structure 

Effectiveness Criterion 

1.1 Statement of Purpose and Administrative Organization - Agency's statement of purpose 
and administrative organization are relevant to the programs, reflect current conditions, 
and support local programs. 

Process Criteria 

1.1.1 Statement of Purpose - Agency has a statement of purpose with related goal state- 
ments that express the principles and concepts adhered to in planning and provid- 
ing programs. 

1.1.2 Organizational Arrangements - Agency's governance and administrative struc- 
ture promotes citizen access and involvement in planning, providing, and evalu- 
ating programs. 

1.1.2a Defined Roles - States clearly the roles, expectations, and levels of 
authority for vocational administrators. 

1.1.2b Commitment - Indicates a continuing conmiitment to providing rele- 
vant vocational education programs. 
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1 . 1 .3 Business Partnerships - Agency's governance, administrative, and advisory struc- 
ture promotes business, indusuy, and labor involvement in planning, providing, 
and evaluating programs. 

1.1.4 Legal Compliance - Agency administrators assume responsibility for ensuring 
that programs meet statutory and regulatory requirements. 

Long-Range Flan 

Effectiveness Criteria 

2.1 Consistency with Commission's Long-Range Plan - Agency decisions on delivery sys- 
tems and im>grams for vocational and technical education are consistent with the detailed 
provisions of the commission's long-range {rian. 

Process Criteria 

2.1.1 Planning System - Agency utilizes a comprehensive planning system that pro- 
vides opportunities for participation and interaction among relevant parties, re- 
lies on and utilizes basic data, includes a comprehensive evaluation system, and 
uses information about current arul potential programs, program participants, and 
employers. 

2.1.2 Stakeholder Involvement - The various parties affected by agency decisions are 
effectively involved in the plaiming process. 

2.1.3 Linkages — Effective communication channels exist between the agency and the 
immediate service area and between the agency and other agencies arid institutions. 

2.1.4 Environmental Scanning - Agency has an ongoing system for collecting, updat- 
ing, analyzing, and integrating relevant information into the planning process. 

2.1.4a World-of- Work/Labor Market Data - Uses current and projected in- 
formation about the general world-of-work and the related ^)ecific em- 
ployment needs in program planning. 

2.1.4b Environmental Data - Uses current and projected employinent, eco- 
nomic, and demographic data about the immediate and extended ser- 
vice areas in the projection of specific employment needs. 

2.1.4c Employee Characteristics - Uses world-of-work leadership profiles in 
the projection of desired employee characteristics. 

2.1.4d Demographic Projections ~ Studies and analyzes projected population 
shifts in terms of their effect on the potential pool of program participants. 

2.1 .4e Potential Staff - Gathers and analyzes data about the age and experience 
proves of program staff and projections about the potential pool of new 
staff from traditional and nontraditional sources in terms of potential im- 
pact on staff availability. 

2.1.4f Funding Levels - Maintains and uses current information on anticipat- 
ed changes in state and fedoal funding levels and/or legislative reauthori- 
zations in program plarming. 

2.1.5 Specific Goals — Agency's plan contains specific goals for public vocational edu- 
cation at all levels that relate to the following: 

Preparation of each graduate for both employment and further education. 
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Accessibility of vocational education to persons of all ages who desire to explore 
and learn for economic and personal growth* 

Projected emfrfoynient opportunities in various vocational and technical fields. 

Supply and demand for a labor force skilled in particular vocational and technical 
areas* 

Technological and economic change affecting Indiana. 

Private vocational education opportunities in Indiana. 

Recommendations for improvement in the state vocational education program. 

Educational levels expected of programs to meet projected employment needs. 

2.L6 Implementation of Plan — Agency inq)lements die long-range state plan for monitor- 
ing vocational education programs under its jurisdiction. 

2.L7 Periodic Updates - Agency updates the plan periodically. 

2.1.8 Dissemination - Agency submits its plan to the commission. 

2.1.9 Annual Report - Agency submits an annual report to the commission that in- 
cludes a section on the contribution that the agency's programs are making to at- 
tainment of the goals in the state's long- range plan* 

2.2 Goal Relevancy — The goals in the agency's iong*range plan for vocational and technical 
education are being used to guide policy decisions at the agency and local levels. 

Process Criteria 

2.2.1 State Agencies - Before fmalizing policy decisions, state agencies consult their 
plans to ensure consistency. 

2.2.2 Program Providers - State agency personnel take steps to ensure that program 
providers consuh the plans of relevant state agencies to determine if contemplat- 
ed local policy decisions are consistent with those plans. 

3.0 Agency Leadership 

Effectiveness Criteria 

3.1 Coordination - Agency works effectively with other state agencies, commissions, and 
organizations that supervise or administer vocational and technical education programs 
in the development of plans or programs to improve the coordination of vocational and 
technical education in Indiana. 

Process Criterion 

3.1.1 Consultation — Personnel in the agency are routinely consulted by other agencies 
in the administration and delivery of vocational education programs. 

3.2 Communication - I^rsons in odier stale agencies with lesponsibilities relate 

al and technica) education are infinmed about the agency's a^jt, authority, and responsibilities. 

Process Criteria 

3/S.l Meetings - Periodic meetings are held with other agencies to discuss issues and 
problems. 

3.2*2 Information - Agency has prepared materials that outline the agency's role, authori- 
ty, and responsibilities. 
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3*3 Professional Relationships - Agency personnel work with program provider personnel 
to improve the administration and delivery of programs. 

Process Criterion 

3.3.1 Assistances - Agency develops effective working relationships with the bodies 
that report to the agency. 

3.4 Program Guidelines - Agency provides program providers with broad guidelines, in- 
formation, and resources for curriculum development and updating. 

Process Criteria 

3.4.1 Competency Erased Instruction - Agency provides information on implementa- 
tion of competency-based instruction. 

3.4.2 Technology - Agency provides information on the use of technology and com- 
puters to achieve curricular goals. 

3.4.3 Integration - Agency provides information on integration of basic and vocation- 
al skills across the curriculum. 

3.5 Program Provider Long-Range Plan Review - Agency provides program providers with 
feedback on their local long-range plans. 

Process Criteria 

3.5.1 Plan Compatibility - Agency reviews program provider plan and checks for com- 
patibility and consistency with state plan. 

3.5.2 Legal Compliance - Agency reviews program provider plans to ensure compli- 
ance with state and federal requirements. 

Fiscal Accountability 

Effectiveness Criteria 

4.1 Priorities and Goals - Agency allocates funds in a manner that encourages programmat- 
ic change, stimulates local investment, improves program quality, and fosters the develop- 
ment and attainment of performance standards. 

Process Criteria 

4.1.1 Incentive Programs - Agency develops incentive programs that provide special 
funding to foster innovation, program improvement, and development and attain- 
ment of performance standards. 

4.1.2 Local Initiatives - Agency collects and analyzes information on program provid- 
er use of incentive funds. 

4.2 Sufficient Funds — The level of funding for programs under the agency's jurisdiction 
is sufBcient to provide adequate financial support for the necessaiy number and perform* 
ance of programs to meet state goals. 

Process Criteria 

4.2.1 Current Programs - Agency consults on a regular basis with program providers 
to determine if the funding level is sufficient for current programs. 

4.2.2 Proposed Programs - Agency requests reports from program providers on the 
funding level required for the introduction of proposed programs. 
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4.3 Funds Allocation - Agency dispenses funds for vocational and technical education pro- 
grams in a manner consistent with the statutoiy requirements of the Indiana General As- 
sembly, fisderal regulations, and the commission's recommendations. 

Process Criteria 

4.3.1 Budgetary Review - Agency prepares proposed summary budget ftom the re- 
quests from the program providers. 

4.3.2 Budget ReoMnmendaticms - Agency makes recommendations to the relevant state 
bodies concerning the af^nropriation of state funds and allocation of federal funds 
for vocational education. 

4.3.3 Accountability for Federal Funds - Agency maintains accountability for all fed- 
eral funds for vocational education under its jurisdiction, including the receipt and 
disbursement of funds. 

4.3.4 Disbursement of Federal Funds - Agency expends and distributes only those federal 
fiinds that have been allocated to it by the commission and the Indiana General 
Assembly. 

4.3.5 Internal Accounting Procedures - Agency has established internal fiscal control 
and accounting procedures. 

4.3.6 Funds Disbursement - Disbursement procedures and records are consistent with 
state and federal requirements. 

4.3.7 Fiscal Audits - Financial audit reports indicate that fiscal procedures are in com- 
pliance with applicable local, state, and federal rules. 

Technical Assistance 

Effectiveness Criterion 

5.1 State-Level Technical Assistance - Agency provides technical assistance on program 
planning, instruction, administration, and evaluation to local program providers. 

Process Criteria 

5.1.1 Technical Assistance Plan - Agency do elops and implements a plan for provid- 
ing technical assistance to local providers of programs, services, aaivities, and 
special projects. 

5.1.2 Technical Assistance Funds - Agency secures funding for staff and related ex- 
penses to provide technical assistance to iocal program providers. 

Program Evaluation 

Effectiveness Criteria 

6.1 Program Evaluation - Agency uses a systematic and comprehensive approach in evalu- 
ating vocational and technical education programs. 

Process Criteria 

6.1.1 Evaluation Procedures - Agency develops procedures for evaluating vocational 
education programs, services, activities, and special projects. 

6.1.2 Methodology - Agency's methodology for local program accountability is based 
on program completer outcomes and labor market requirements. 
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6.1.3 Outcome Measures — Agency develops a management infomiation system to mon- 
itor program outcomes in the following target areas: 

Program enrollment and completion 

Demonstrated competency of program completers 

Af^lication of skills by program completers 

Po£tprogram formal education 

Gainful employment 

Employment related to training 

Income of program participants 

Employment stability 

Program completer satisfaction 

Employer satisfaction with performance 

6.1.4 Aggregate Effectiveness - Agency uses aggregate data from its management in- 
formation system in evaluating the extent to which vocational education programs 
are contributing to attainment of the goals in the state's long*range plan for voca- 
tional and technical education. 

6.2 Utilization of Data - Agency's data base is used in making both program and public- 
policy decisions about vocational and technical education at the agency and local pro- 
gram provider levels. 

Process Criterion 

6.2.1 Design of Information System - Agency designs and implements a compatible 
data system with other agencies that provides information about the performance 
of individual program providers and overall current educational delivery system. 

Articulation 

Effectiveness Criterion 

7.1 Institutional Transition — Program participants have only minimal redundancy, logisti- 
cal, or program-planning problems making the transition among and between secondary 
and post secondary vocational and technical education programs and college degree 
programs. 

Process Criteria 

7.1.1 I>evelopment of Transition Plan - Agency develops a plan for improving the pro- 
gram participants' and program content tra; ition between secondary and post- 
secondary vocational education programs. 

7.1.2 Implementation of Transition Plan — Agency implements a plan for improving 
the program participants* and program content transition between secondary and 
postsecondary vocational education programs. 

7.1.2a Adult Education Task Force - Works with the task force to study the 
adult education continuum of technical education and training. 

7.1.2b Reconunendations for Adult Education - Makes recommendations con- 
cerning the adult education continuum of technical education and train- 
ing and degree programs. 
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7«L3 Assessment of Remedial Education - Where ap|4icable, agency assesses the deliv- 
ery of remedial education by postsecondaiy institutions. 

8.0 ProfeaskMial Peraonnel Dereiopiiient 

Effectiveness Criteria 

8*1 Teacher Certification - The agency guidelines for vocational teacher ceiUfication or recer- 
tification are consistent with the commission's recommendations. 

Prociess Criterion 

8.1.1 Professional Development - Where applicable, agency maintains a comprehen- 
sive professional development program. 

8,1,1a Staffing Flexibility - Studies the extent to which demonstrated teach- 
ing competence should permit flexibility for experts outside the voca- 
tional arKl technical education structure to teach special classes. 

8.1.1b Modifications of Certification - Solicits reconmiendations from outside 
the field of vocational/regular education about modifying vocational teach- 
er certification or recertification. 

8.2 Sufficient Staff - The number of qualified vocational and technical education teachers 
is sufficient to staff the programs in the state. 

Process Criterion 

8,2,1 Recruitment and Staff Development - Where applicable « agency maintains a teach- 
er recruitment and staff development program to foster improvement in the voca- 
tional and technical education delivery system. 

8.2.1a Personnel Policy - Adopts and implements a personnel policy to at- 
tract and retain highly qualified vocational teachers. 

8*2. lb Salary Schedules - Encourages program providers to adopt salary sched- 
ules that recognize years of occupational experience, including time spent 
in business/industry partnerships, as well as teaching experience, ex- 
pertise, and formal educational preparation. 

8.2,1c Fellowships and Grants — Develops and implements a program of grants 
or fellowships to local institutions or individual teachers for the purpose 
of permitting professional updates through a variety of high-potential 
techniques. 

8.2. Id Private Sector Incentives - Develops reconmiendations regarding in- 
centives available to businesses and industries to encourage their col- 
laboration in providing quality occupational internships for teachers. 

8,2, le Flexible Staffing Practices — Develops statewide guidelines for staff- 
ing policies and practices in vocational education that will increase the 
likelihood of teachers remaining current in their occupation and will en- 
courage the hiring of qualified part^ime instructors and/or required, regu- 
lar, paid business/industry internships for teachers. 

8.2. If Staff Development — Develops statewide guidelines for the preparation 
of new teachers and upgrades existing faculty to enhance the instruc- 
tional capabilities for the reinforcement of basic academic skills In voca* 
tional education. 
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83 Teacher Skill Improvement - Agency works with program providers in implementing 
the commission's long-range plan for incorporating updated technological content and teach- 
ing skills into vocational and technical education programs* 

Process Criteria 

8.3.1 Technological Update - Agency recommends that local program providers as- 
sess the need for technological update and develop a plan to address this need. 

8.3.2 Strategy on Technology - Agency promotes a comprehensive strategy concern- 
ing technology and education. 

8.3.2a Teacher Participation - Where applicable, develops and implements a 
plan to encourage teachers to participate in all phases of the process of 
technological update and to incorporate updated technological teaching 
skills into teaching programs* 

8.3.2b Private Sector Participation - Develops and implements a plan to en- 
courage private-sector participation in all phases of the process of tech* 
nological update. 

8.4 Tax Incentives for Teachers - Where applicable, the agency makes legislative recom- 
mendations to the conmiission for a professional development credit system and tax in- 
centives for teachers. 

Process Criterion 

8.4.1 Professional Development Tax Incentive — Agency studies and niokes recom- 
mendations concerning a professional development credit system and tax incen- 
tives for teachers. 

Partnerships 

Effectiveness Criteria 

9.1 Partnership Programs - Agency develops and maintains partnership programs between 
and among business, industry, labor, government, and education. 

Process Criteria 

9.1.1 Stronger Relationships - Agency develops and implements a plan to strengthen 
relationships between and among those involved in education and employment 
training. 

9.1.2 Expanded Partnerships - Agency expands partnerships between and among busi- 
ness, industry, labor, govenunent, and education. 

9.2 Technology Education Curriculum - Representation from business, industry, labor, 
government, and education assists in the design, development, and implementation of 
the technology education curriculum. 

Process Criterion 

9.2.1 Private/Public Participation - Agency participates in meetings for the purpose 
of working with representatives from business, industry, labor, govenmient, and 
education to design, develop, and implement the technology preparation cur- 
riculum. 
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System Effectiveness and Process Evaluation Criteria for 
Indiana Commission on Vocational and Technical Education 

The foUowing list of effecdveness and process evaluation criteria for the overall systems evaluation 
of the Indiana Ccmimission on Vocational and Technical Education has been developed as a response 
to task 5 in the project contract. The Indiana Commission on Vocati(Mial and Technical Education 
is the state-level coordinating body for vocational and technical education. The principal sources 
for these criteria were the commission's long-range plan, related statutes, and commissioned studies. 
The criteria have been divided into nine broad areas: 

1.0 Long-Range Plan 

2.0 Duties of the Commission 

3.0 Fiscal Resources 

4.0 Program Delivery 

5.0 Articulation 

6.0 Professional Personnel Development 

7.0 Partnerships 

8.0 Technical Assistance 

9.0 Vocational and Technical Education Evaluation Criteria 

'^utsc evaluation criteria are designed to focilitate tracking the progress of the commission in im- 
plementing those strategies outlined in the conunission's long-range plan and to permit refinement 
of specific evaluation criteria as strategies are implemented. The process criteria may be used by 
the commission in assessing the extent to which an effectiveness criterion has been attained and in 
determining responses to identified problems. 

1.0 Long-Range Plan 

Effectiveness Criterion 

1.1 Commission's Long-Range Plan — Commission adopts and updates a long-range plan 
for the provision and continuous improvement of vocational and technical education in 
Indiana. 

Process Criteria 

1.1.1 Development of Plan - Commission develops a long-range state plan for a com- 
prdiensive vocational education program. 

1.1.2 Implementation of Plan - Commission implements a long-range state plan for 
a comprehensive vocational education program. 

1.1.3 Periodic Updates - Commission updates the plan periodically. 

1.1.4 Dissemination — Commission disseminates revisions of the plan to appropriate 
parties as prescribed in Indiana Public Law 217, sec. lO.a (1987). 

1.1.5 Specific Goals — Conomission's plan contains specific goals for public vocational 
education at all levels that relate to the following: 

Preparation of each graduate for both employment and further education. 

Accessibility of vocational education to persons of all ages who desire to explore 
and Icam for economic and personal growth. 

Projected employment opportunities in various vocational and technical fields. 
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Supply and demand for a labor force skilled in particular vocational and technical 
areas. 

Technological and economic change affecting Indiana. 

Private vocational education of^rtunities in Indiana. 

Recommendations for improvement in the state vocational education program. 

Educational levels expected of programs to meet projected employment needs. 

1.1.6 Annual Report — Commission submits an annual report that includes a section 
on the contribution that vocational education is making to attaining the goals in 
the state's long-range plan. 

2.0 Duties of the Commission 

Effectiveness CtJeria 

2.1 Enactment of Reconmiendations - The Indiana General Assembly enacts the conunis- 
sion's recommendations concerning the development, dupUcation, and accessibility to em- 
ployment training and vocational and technical education on a regional and statewide basis. 

Process Criteria 

2.1.1 Presentation of Recommendations - Commission makes recommendations to the 
Indiana General Assembly concerning the development, duplication, and accessi- 
bility of employment, training, and vocational education on both a regional in- 
trastate and a statewide basis. 

2.1.1a Current Missions ~ Studies the current missions of all components of 
public education in Indiana and makes recommendations for changes 
where ^ropriate. 

2.1.1b Private Services - Studies the impact of all private vocational educa- 
tion services and makes recommendations where appropriate. 

2.1.1c Access - Develops and recommends policies to encourage increased 
participation of minority and other undenepresented groups in the state's 
vocational education system. 

2.2 Coordination - Commission works effectively with state agencies, state commissions, 
and community-based organizations that supervise or administer vocational and technical 
education programs in the development of plans or programs to improve the coordination 
of vocational and technical education in Indiana. 

Process Criteria 

2.2.1 Consultation - Commission consults with all state agencies, commissions, or or- 
ganizations that supervise ov administer vocational education programs on ways 
to improve the coordination of vocational education in Indiana, including the fol- 
lowing: 

Department of Commerce 

Job Training Coordinating Council 

Department of Employment and Training Services 

A private industry council (as defined in 29 USC 1501 et seq.) 

Department of Labor 

Conunission on Proprietary Education 
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Commission for Higher Education 
State Board of Education 

2.2.2 Adult Basic Skills - Commission supports die Adult Basic Skills Oversight Com- 
mittee regarding delivery of basic academic skills for adults. 

2.2.3 Coa^)etency-Based Education - Commission provides guidelines for competency- 
based education. 

2.2.4 Agency Plans — Conmiission develops a system to review and make recommen- 
dations concerning vocational education plans submitted by Department of Edu- 
cation, Commission for Higher Education, and Department of Employment and 
Training Services. 

2.2.5 Review of Agency Recommendations - Commission implements a system to re- 
view and make reconmiendations concerning vocational education plans submit- 
ted by Department of Education, Commission for Higher Education, and 
Department Ot Employment and Training Services. 

2.2.5a Consistency - Reviews and prepares reports on the plans of each agen- 
cy to ensure that they are consistent with the commission's long-range 
plan for the state. 

2.2.5b Compatibility - Reviews and prepares reports on the plans of each agency 
to ensure that they are compatible with each other. 

2.2.5c Nonduplication - Reviews and prepares reports on the plans of each 
agency to ensure that adequate efforts are being made to avoid duplica- 
tion in delivering services by the three agencies. 

2.2.6 Reports to Legislature - Commission reports to the Indiana General Assembly 
on conclusions and recommendations concerning interagency cooperation, coor- 
dination, and articulation of vocational education and employment training. 

2.2.6a Development of Strategy - Develops a strategy for intrastate region^.! 
communication, coordination, and cooperation. 

2.2.6b Implementation of Strata - Implements a strategy for intrastate regional 
communication, coordination, and cooperation. 

2.2.6c Interagency Partnerships - Develops a plan to strengthen and expand 
partnerships with any state agency, conunission. or organization that su- 
pervises or administers vocational education programs. 

2.2.6d Expansion of Partnerships — Implements a plan to strengthen and ex- 
pand partnerships with any state agency, commission, or organization 
that supervises or administers vocational education programs. 

2.3 Federal Funds - Commission receives and allocates the federal funds available for voca- 
tiotud and technical education. 

Process Criterion 

2.3 1 Procedures for Agreements - Commission develops and utilizes procedures for 
entering into agreements with the federal govenunent that are required as a con- 
dition of receiving funds under the Vocational Education Act. 
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2.4 Communication - Commission has systematic and purposeful communication channels 
with business, industry, and labor organizations that cerate vocational and technical edu- 
cation programs. 

Process Criterion 

2.4.1 Intrastate Coordination — Conunission establishes a regional inti state planning 
and coordination system for vocational education and employment training. 

2.4.1a Appointment of Advisory Councils - Appoints advisory councils as 
needed. 

2.4.2b Implementation Contracts - Contracts for services as necessary to im- 
plement the plarming and coordination system. 

2.4.3c Private Programs - Provides information and advice on vocational edu- 
cation to business, industry, or labor organizations that operate a private- 
sector job training program. 

3.0 Financial Resources/Responsibilities 

Effectiveness Criteria 

3.1 Priorities and Goals -> Commission allocates funds in a marmer that encourages program- 
matic change, stimulates local investment, improves program quality, and fosters the de- 
velopment and attaitmient of performance standards. 

Process Criteria 

3.1.1 Guidelines and Priorities - Commission establishes planning guidelines and priori- 
ties for incentive programs that will be used in the allocation of discretionary funds. 

3.1.2 Lx)cal Initiatives - Commission collects and analyzes information on program 
provider use of incentive funds. 

3.2 Funds Allocation - The Indiana General Assembly enacts the commission's recommen- 
dations for allocating funds for vocational and technical education programs. 

Process Criteria 

3.2.1 Finance Study - Commission conducts a comprehensive study of the financing 
of vocational and technical education that includes programs costs, levels, adminis- 
tration, support, and all related resources. 

3.2.2 Finance Study Report - Commission prepares a report based on the comprehen- 
sive study of the financing of vocational and technical education. 

3.2.3 Strategies and Resources — Conmiission presents a report to the Indiana General 
Assembly on the strategies and required resources for strengthening vocational 
and technical education in the state. 

3.3 Private-Sector Tax Incentives — The Indiana General Assembly enacts legislation to provide 
tax incentives and tax credits to encourage involvement of private-sector firms in voca- 
tional and technical education programs. 

Process Criteria 

3.3.1 Tax Incentive Recommendations — Commission malces reconunendations to the 
Indiana General Assembly concerning tax incentives and tax credits. 

3.3.1a Coordination - Specifies ways to in^rove business coordination and 
investments in vocational/technical education. 
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33.1b Incentives - Identifies rewards that could be given to businesses and 
industries that provide the equipment necessary for current occupation- 
al technology. 

3.3.1c Internships - States incentives diat could be used to encourage businesses 
and industries to collaborate in providing quality internships for teachers. 

3.3. Id ProfessionaUzation of Teachers - Indicates steps dmt could be taken 
to encourage teachers to participate in profiessional development activities. 

3 .3.2 Support for Partnerships - Commission disseminates infonnadon to agencies and 
program providers about tlie availability of financial incentives from federal, state, 
and other sources ^xat provide support for partnerships. 

3.4 Foundation - Cc mmission has established and maintains a nonprofit foundation for the 
improvement of vocational and technical education in Indiana. 

Process Criterion 

3.4.1 Membership - Conunitsion identifies intertsted citizens and businesses and takes 
the necessary legal and procedural steps to create a nonprofit foundation of the 
improvement of vocational and technical education in Indiana. 

3.5 Fiscal Audits — Financial audit reports indicate that fiscal procedures are in compliance 
with applicable local, state, and federal mles. 

Proctss Criteria 

3.5.1 Budgetary Review - Commission reviews tht; legislative budget requests for voca- 
tional education prepared b)' Department of Education, Commission on Higher 
Education, and Department of Employment and Training Services. 

3.5.2 Budget Recommendations ~ Commission makv?s reconunendations to the budget 
co nunittees of the Indiana General Assembly concerning die appropriation of state 
fmd& and allocation of federal funds for vocational education specifying mini- 
muin funding levels required by 20 USC 2301 and 29 USC 1533, categories of 
expendimres and the distribution plan or formula for secondary schools, and cat- 
egories of expenditures for each state postsecondary institution. 

3.5.3 Accountability for Federal Funds - Commission maintains accountability for all 
federal funds tor vocational education under 20 USC 2301 and 29 USC 1533, 
including the receipt and disbursement of funds. 

3.5.4 Disbursement of Federal Funds - Commission expends and distributes only those 
federal funds that have been allocated by the Indiana General Assembly. 

3.5.5 Audit Reports - Audit reports indicate minimal negative findings in the handling 
of funds. 

4.0 Program Delivery 

Effectiveness Criteria 

4.1 Program Provider Evaluation - Conunission's evaluation and data system provides and 
uses information to improve the effectiveness of individual program providers. 

Process Criteria 

4.1.1 Design of Information System — Commission designs an evaluation and d£.ta in- 
formation system that provides information about the performance of the individual 
program providers. 
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B 4.1.2 Implementation of Information System - Commission implements an evaluation 

and data system that provides information about the performance of the individu- 

■ al program providers. 

4.2 Statewide Program Evaluation - Commission's evaluation and data system provides and 
H utilizes statewide information on q)ecific occupational training programs to improve the 

■ statewide effectiveness of these programs. 

Process Criterion 

B 4.2.1 Statewide Data Compilation - Commission compiles, reports, and analyzes data 

m on statewide compilation of program information by specific occupational train- 

ing programs. 

I 4.3 System Program Effectiveness - Commission's evaluation and data system provides and 

utilizes information to improve the effectiveness of the state's comprehensive vocational 
m and technical education delivery system. 

■ Process Criteria 

4.3.1 Compatible Data Systems - Commission develops and implements compatible 

■ data systemf for the delivery of occupational information and career information 
• to local delivery systems in a timely and cost-effective manner. 

■ 4.3 J Trends in Remediation — Conmiission provides information and makes recom- 
I mendations concerning trends related to the need for remediation of basic skills. 

M 4.3.3 Counselor Preparation - Commission studies and reports areas of concern regard- 

I ing counselor preparation, credentialing, and professional development. 

4.3.4 Economics Education - Commission conducts a study and develops a strategy 

■ to determine when, where, and how to include in vocational education an under- 

■ standing of economics and the economic system that reflects the state, national, 

and world economy. 

■ 4.3.5 Program Participant Services Plan - Commission develops and implements a plan 

to improve guidance, counseling, placement, and career development services. 

I 4.3.6 Assessmen t of Remedial Education - Commission assesses the delivery of remedial 

education by postsecondary institutions. 

I 4,3.7 Recognition Program - Commission recognizes and honors exemplary career pro- 

grams and services on an aimual basis. 

■ 5.0 Aitkulatioii 

■ Effectiveness Criterion 

5.1 Institutional Transition - Commission has taken necessary steps to ensure that program 
I participants have only minimal redundancy, logistical, or program-planning problems mak- 

■ ing the transition among and between secondary and postsecondary vocational and tech- 
nical education programs and college degree programs. 

I Process Criteria 

5.1.1 Develqnnent of Transition Plan - Commission develops a plan for improving 

■ the transition of program participants and program content between secondary and 

■ postsecondary vocational education progmns. 
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5. 1 .2 Implementation of TransitioD Plan - Commission implements a plan for improv- 
ing the transition of program participants and program content between second- 
ary and postsecondary vocational education programs. 

5.1^ Adult Education Task Force - Appoints a task force to study the adult 
education continuum of technical education and training. 

5. 1 .2b Recommendations for Adult Education - Makes recommendations con- 
cerning the adult education continuum of technical education and train- 
ing and degree programs. 

6.0 Professkmal Personnel Deveiopnient 

Effectiveness Criteria 

6.1 Teacher Certification - State system for certifying or recertifying vocational teachers 
is consistent with the conunission's recommendations. 

Process Criterion 

6.1.1 Professional Development - Conmiission maintains a comprehensive professional 
development program and has adopted a recommended set of standards for cer- 
tifying teachers. 

6.1.1a Staffing Flexibility - Studies the extent to which demonstrated teach- 
ing competence should peniiit flexibility for experts outside the voca- 
tional and technical education structure to teach special classes. 

6.1.1b Modifications of Certification — Solicits recommendations from outside 
the field of vocational/regular education about modifying vocational teach- 
er certification or recertification. 

6.2 Sufficient Staff - The number of qualified vocational and technical education teachers 
is sufficient to staff the programs in the state. 

Process Criterion 

6.2.1 FCecruitment and Staff Development — Commission maintains a teacher recruit- 
ment and staff development program to foster improvement in the vocational and 
technical education delivery system. 

6.2.1a Persomiel Policy - Adopts and implements a personnel policy to at- 
tract and retain highly qualified vocational teachers. 

6.2.1b Salary Schedules — Encourages local school districts to adopt salary 
schedules diat recognize years of occupational experience, including time 
spent in business/imlustry partnerships, as well as teaching experience, 
expertise, and formal educational preparation. 

6.2.1c Fellowships and Grants - Develops and implements a program of grants 
or fellowships to local institutions or individual teachers for the purpose 
of permitting professional updates through a variety of high-potential 
techniques. 

6.2.1d Private Sector Incentives - Develops reconmiendations regarding in- 
centives available to businesses and industries to encourage their col- 
laboration m providing quality occi^ntional internships for teachers. 

6.2. le Flexible Staffing Practices — Develops statewide guidelines for staff- 
ing policies and practices in vocational education that will increase the 
likelihood of teachers remaining current in their occupation and will en- 
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courage the hiring of qualified part- time instructors and/or required, 
regular, paid business/industry internships for teachers. 

6.2. If Staff Development - Develops statewide guidelines for the preparation 
of new teachers and provides for upgrading existing faculty to enhance 
the instructional capabilities for the reinforcement of basic academic skills 
in vocational education. 

6.3 Teacher Skill Improvement - Program providers implement the commission's long-range 
plan for incorporating updated technological content and teaching skills into teaching 
programs. 

Process Criteria 

6.3.1 Technological Update - Commission recommends that local program providers 
assess the need for technological update and deve'op a plan to address this need. 

6.3.2 Strategy on Technclog)' - Commission promotes a comprehensive strategy con- 
cerning technology and education. 

6.3.2a Teacher Participation - Develops and implements a plan to encourage 
teachers to participate in all phases of the process of technological up- 
date and to incorporate updated technological teaching skills into teach- 
ing programs. 

6.3.2b Private Sector Participation - Develops and implements a plan to en- 
courage private sector participation in all phases of the process of tech- 
nological update. 

6.4 Tax Incentives for Teachers - The Indiana General Assembly accepts the commission's 
recommendations for a professional development credit system and tax incentives for 
teachers. 

Process Criterion 

6.4.1 Prolessional Development Tax Incentive - Commission studies and makes recom- 
mendations to the Indiana General Assembly concerning a professional develop- 
ment credit system and tax incentives for teachers. 

Partnerships 

Effectiveness Criteria 

7.1 Partnership Programs - Commission develops and maintains pannership programs be- 
tween and among business, industry, labor, government, and education agencies. 

Process Criteria 

7. 1 . 1 Stronger Relationships ~ Cranmission develops and implements a plan to AOengthen 
relationships between and among those involved in education and employment 
training. 

7.1.2 Expanded Partnerships - Commission expaiKls partnerships between and among 
business, industry, labor, government, and education. 

7.2 Technology Education Curriculum - Representation from business, industry, labor, 
government, and education provides assistance in the design, development, and implemen- 
tation of the technology education curriculum. 
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Process Criterion 

7.2*1 Private/Public Paiticipati(» ~ Commission organizes meetings for the purpose 
of W(n1dng with representatives from business, industry, labor, government, and 
education to design, develot> and im|rfement tfie technology education curriculum. 

8.0 Technical AastsUnce 

Effectiveness Criterion 

8.1 State-Level Technical Assistance ~ Local program providers receive technical assistance 
on program planning, instructron, administration, and evaluation. 

Process Criteria 

8.1.1 Technical Assistance Plan — Commission works with state agencies in the de- 
velopment of a plan to provide technical assistance to local providers of programs, 
services, activities, and special projects. 

8.1.2 Technical Assistance Funds — Commission and agencies secure funding and im- 
plement the plan to provide technical assistance to local providers of programs, 
services, activities, and special projects. 

9.0 Vocational and Technical Education Evaluation Criteria 

Effectiveness Criteria 

9.1 Program Evaluation ~ Commission uses a systematic and comprehensive approach in 
evaluating vocational and technical education programs. 

Process Criterion 

9.1.1 Evaluation Criteria - Commission establishes criteria for evaluating vocational 
education programs, services, activities, and special projects. 

9.1.1a Evaluation Study — Conducts a study on evaluation of vocational edu- 
cation and makes recommendations concerning the following 

Procedures for locally directed evaluations. 
Methods to validate skill requirements. 

Methods of data coUecti'^n, storage, and analysis, including the feasi- 
bility of c(Hnputer usage. 

Procedures for coordinating data bases and program evaluation systems. 

Criteria for evaluating programs and services. 

Criteria for evaluating and analyzing the effectiveness of the evaluation 
systems being used. 

9.1.1b Methodology for Accountability - Develops a methodology for local 
program accountability based on program participants outcomes and la* 
bor market requirements. 

9.1.1c OutoHne-Based Evaluations - Conducts a study and prepares report on 
the effectiveness of outcome-based program evaluations. 

9.1. Id Counseling Programs - Makes reconunendations concerning evalua- 
tive criteria concerning the effectiveness of counseling programs. 

9.2 Utilization of Data - Commission's c(nnprehensive data base is used to make both pro- 
gram and public-policy decisions about vocational and technical education at the state 
and local levels. 
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Process Criteria 

9.2.1 Data Systems - Commission develops and constructs data systems for program 
evaluation and delivery of services. 

9.2.2 Planning Data - Coromission collects, analyzes, and makes appropriate data avail- 
able for planning puiposes. 

9.2.3 Employment Opportunities - Commission prepares reports that project employ- 
ment opportunities in various vocational and technical education fields. 

9.2.4 Specific Employment Needs - Conunission ainducts a study and prepares reports 
on the supply and demand for a labor force stalled in particular vocational and 
technical areas. 

9.2.5 Environmental Change Smdy - Commission conducts a study and prepares an 
analytical report on the technological, demographic, and economic changes af- 
fecting Indiana. 
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